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This review, of selected study designs, the first 
voljume of a two-volume resource , is intended to advise state » v 
administrative prbgram leaders and others with program evaluation >> 
responsibilities. of how to. design ^ studies of cooperative extension 
programs. Three kinds of informatipn are presented in the volume ■ s. 
thr^e ^parts. The first part introduces the nature and purpose of ^ 
stAidies of extension program results and places the design of studies 
into pef^ective. It reviews evaluation users, dimensions, general 
procedures, 'and standards. Alternatives to designing studies of ^ 
extension program results are cited.in pa^st 2. Four bas;ic stiioy 
designs used in evaluating extension program results are examined 
((1) survey (ex-post .facto) , (2) timq-series, (3) compariso'n group, 
and (4) field experiment) , and their relative suitability is linked 
to practical considerations based on the evaluation situation. This 
part also lists 10 specific facets of the study' designs and 
illustrjates howi each may lead to alternative ways of constructing a 
design for a study. Part- 3 provides a brief review of the conditions 
that "lend themselves to alternative study designs and suggests . 
contexts in which this pesource publication should be viewed. 
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PREFACE • 

For accountaiDllitv and administrative , decisions, there Is, 
.Increasing need for State Cooperative Extension program evalu^tipn 
studies which are credible to ^ State and Federal legislators and 
executives, university, leacfefrs and Fxtenslon administrators. This need 
is made apparent by the*" recently adopted Extension Accountability and 
Evaluation- System. The system calls for the .Extension Service, USDA, to 
provide staff development and technical assistance to State Extensi^on 
Services', studies on the inpyts, opexa^tions and ' impacts of Extension 

programs. In January 1983, Extension Service, USDA, entered into, a 

- / 

cooperatlA^e agreement with the Maryland Cooperative Extension Service, 

through the University 'of Maryland's Department of Agricultural and 

Extension Education, to develop and publish a^resourcj3 publication on 

designing studies to evaluate ' the » results of Cooperative Extension 

programs. ^ sj • ' . 

The purpose .of this two-volume resource is to. advise state 

adminlRtrative program leaders and others with program evaluation 

responsibilities ,how to* design studiofi of program results that are 

accurate and reliable. ^Thin" resoulrco may nlso aid in the development of 

* f 

more uniform standards for progratn /evalufltion within Cooperntive 
Extension. . • % ^ ' 

Volume • !• of t.hc Resource reviews sclc^ctod study dceigns. TUesc 
illj^Btratc dlff crept: approaches to 'oxnmlnln/> the ^oxtoiit to which 
cHontelo behavior or status can be ^.nttributcd to an Kxtcnnion program. 
In ordeV to show the fonoibillty of using a variety of cvalurttlve atudy 
designs to examine progrnm roaults, Volume II includes nbetratftfl of 42 
otudica m examplca of thotiu study doolgno. The nlJottactd, were nelGcttd 
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to ' represent Extension programs in agriculture, natural resources^ 
community development, 4-H youth,, Tidme ..economics and . ov^all county 
Extension programs. Each study exemplifies one cr more of four basic 
st^dy dfeslgns: (1) survej^ (ex-post, facto), (2) time-series, (3) com- 
parisor group", 'Ox\ (4) field experiment. These study designs and the 

. \ r.. t, ' ' ' 

example studies should not limit future evaluation' efforts. They are 

•• ' ^- ■ . ■ ■ ^ ■ * 

intended 't6 provide State Ext^enfeion Services with background kno^edge 

' '\ ' ^ * ■ , " " 

,for designing • studies applicable to tlieir own ^evaluation interosts and 

' )/ ' . # ■ 

needs. ^ ' . 



■ J.- 
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PART I - BACKGROUND AND APPROA'CH 
CHAPTER J : INTRODUCirON , 
Purposes of the , Resource : ^ 

■ . \ 

The publication of this resource is In partial response to" throe 

documents: (1) "Rcp(\rt of the National . T^isk Force on Extension 
• . •* . • 

'Accountability and Evaluation Syfitem" (1981), approved by the Extcnr.lon 

\ 

Committee on Organization and Folic'y (KCOP); (?.) ''The Comptrollor 
General's Report to ' the Congress on **Cooperat ive Extension Sorvice' r. 
Mission and Federal I Role Need, Congressional Clarification'* (1981); and 
(3) Program Evaluation in Exten(\ion , a *198() report on the status of the 
^rganizalflon for and prnctlce of program evaluation by aV^^ State 
KxtenMlon ServiccfU 

The '^Report of the National Task Force nt5 Kxtens ton . Accountab n Uy 
and Evaluation System** includes the recommendations that: (1) State 
ExtenHion Servlcerj undertake Indepth studies of the inputs, operations 
and Impncts of M«*b?cted proKrams, primarily in orcfer to meet nt^Ue <ir 
multl-fitate need'rt U\t /iccotnit abll ity and evaluation; and (2) Ejktrnrlon 
Service, USt)A, ' provide staff ttrvelopwcnt attd technical afi«lat?ince to 
State Extenolott Services In the planning and condueting of such r.tuc^leru 
( The Ceneral AccountluR Offlr.e'n '*Cf>^operat Ive Exten«:h)n Servldt^'a 
Mlr^Mlotj and Federal Role Nerd (;oti|j;renrnon«l Clarlf iK^itton** rerotmRendn 
that the Secretary of Agriculture afisume leadernhir* with F.C01\ for 
developing amf Implement lag a ^niiform evaluation rtyetem for Cooperative 
Ext^naion* Including elearly defined evaluation rttandatd^^, Thir^ 
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resource uses a number of technical standards for conducting ntudles on 
Fxtcnr.ion program results. These ad hoc standards rny Jead to further 
discussion, within Fxtenslon regarding appropriate. titandards for 
ev'nJuation studie.s within the Extension Accountc^bl 1 ity and. F.valuatton 
Systen, . . , 

^naUy» the 1980 national study conducted by West Virginia 
UnlveV.sity reports that: (1) 60 percenH^ of the Str^ e ,rxtcMu; ion Serviceft 
had not assigned specific responsibility for evaluation to any 
particular fitaff roen^ber; (2) almost two-thirds of state pn^gram leaders 
and almost ihree-f ourth.« of county agent« constc^cred thenfu* Ives prepared 
only to "some extent'* or ti> a *\small extent** to adec^uately plan; Imple- 
ment , or report p^rogram ev.1l^tn^ions; and {'^) 85 percent oi State Kxten- 

nion d 'r*-!- tor;;" 1" I'll •••tnll.Trlv nbouL their j;t;iffs' Ir.Kloqu.itc prcp.ir.»tlii;i 

/ . ' 

ta conduct pr«)grnm evaltiatlons. ' 

The ma lor it V (^1 Slate llxtennlon Si^rvicen tiave no hirU(^rv of con- 
titiCting r;tndlev; ol program reHUltyi with the ri^ior oxpeetrd l>y deci?Uon~ 
makers (or, more likely, their staffs) at State and Federal Irvelr;. 
Typtc.iUy* stateii do not h.ivr rvalti.'Uion specta Mfi'Tn to ntlmulate, 
ci»hdt»ct or give ndvlce on ?itiidloM of ptogran tcMUiltn. Thtin, the 
l-Ktenrdun .Srrvlre, I'SDA, entered luln n cof^pc^iat Ive agM-'ement with tUv 
Marvhuul ('r>c>pct at Ive Kxt r^tuil^^^ Service to prodtfre a rertourrc pubUcati<«n 
that will asjilrtt program leMdei r4»;and ripoHa llntn In doMlgnlng etudien of 
program lermlfn which Wlfl he red 11) ff^ Mtudv Ufr^ern fti thn Cpoptifat t ve 
IrHtrUrildU fauvlri' au<l elM^whetr. ' - * 

the rotr and ev^iluatlve eSpctt^ne of KHlm^iion pfogtan lefiderrt an<l 
tjpeelnUrttn will vai'v from ntudv li^ rUudv* However, thirt' tefitnirei* 
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publication .should help, prograa lenders ^and 5p**clnUnt 5, pcrforn the 
following roles with rcf^pect to studies of Fxtcn*ilon pr^ugraa refsuU?;: 



• Advise Kxtenftion cvnluntlon spcclnil fit?; as Chfv design ntudlr^ 
of program results, ' 

• Serve on stot_Tlng connltteef? where progran and cv/iluntion 

personne 1 ' have joint responsibi 1 ity for designing nnd 

conducting a f-tudy of pr0gr.1t?! reflultfu ^ 
'1 • ■ 

*• . • * 

• Kormu\ntc and manage contfflctH or coopi^mc Ive .ngreen^nt ?» , with 

protcr;Blonal . rVrilunitor?? oiitnldc Kxtcnnlon, to fttudy rcTnultn of 
i-.x tonnlon progrnmf^, 

• Penipn nnd conduct ??tud fen of prngrnni reriultn In con-^ul r ion 
with Kxtcnr?lon or non-Hxtcn?rl'on cvnVvntton nprc tnU ntn, 

Thl?r r«M;ourcc publlcntion containn exrtmpte?i of prevltnifi ntudlcf^ 

which dcnonntrat** fo.ifUblo unyn to .-^ricer t;t In rcnultrs of Fxtrnnlon 

m 

pro^r/tn-i lining fovir ftelcctcd ruudy derilgTUt — the riurvry, tlmr-Nerlen, 
» y 

oompar 1 fxni grotip nnd flfld expi*rlment. Th«* rrrtouroe m^iy .tlf^o provt* of 

■f 

'value to fivnlUftilon npt^o i A 1 1 rst « : (1) fifi rt irf>teaOf* ro. prevlon.-i ri<trn« 
nlon ntudlen; Kl) X^^x Inri^rvLce trrttnlng ^^nd rlAns^room e^tviCAt Ion; nnd 
(1) lor nynt rnntlcnlly reviewing choice^ in der?lgnlnR -i rstudv f.Men- 
riion progr.'iP r^^r^tiltM. Fv.-tluntion riperlnl l*itfi r.iy gnln furiti^^r .npprerln- 
tion of option^*! ttiethotifi drmonrurnted hy tht^-^^ prtril nMutlt^w mid ht^v 

tho«e mny hr^ ?M»plf^4 rtt Ir^nnt Jn pnrf, tn rurrent ev<ilurtlitn rfforta. 

•11 

Mor<?ov^r. \ teflourcf^ volumes qhooji*! f?tlh^inc«? ^ommnnli rti ton fimong 

' \ 

I'xt^nrtlon progr^nr" rtt^if f * unlvernity iic4di?mie ntnff, ^ind non-univer«H_y 
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Guide C<5 Using this Resource 



We /ntend for mSSTONING STUDIES OF EXTENSION PROGRAM RESULTS ^o - 
.provide three kfnds of information.,^ These are presented in the three, 
parts of.VolMme I. i . / 

- Part 'I i^roduces ' the nature and purpose t)f studies of Extension 
program results and places the design of steadies into perspective. Part . 
I reviews evaluation usfers, ^dimensions, general procedurejs / and 
standards. . . ' . . / 

Part li cites alternatives to' desigi^i^ig studies of Extension 



program results. These results may^be (a) educational (know! 



edge, 



attitudes, skills and aspirations); (b) practices based on the use. of 
new learning, or (c) impacts^-the ^ eventual economic, socia l or 
environmental results of new learning and practices. Part II examines 
four basic study designs used in evaluating Extension program results, 
and then links their relative suitability to practical considerations 
based on the evaluation situation!^; This part also lists ten spec^ic 
facets of the study designs and illustrates how each of these facets^may 
lead to alternative ways of constructing a design for a study, \ For 
consultation, the facets are referenced to the abstracts o| previous 
.studies of program results in Volume II. These abstract.ed studies are 
dimply examples of .the^ use of the four basic stody .designs — not 
necessarily exemplary studies. . " ' 

Part III provides a brief review of the conditions which lend 
themselves to alternate study designs, and suggests contexts in which 
the resource publication should be viewed. «^ V 
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. ' For ease In comparing the Volume I text on dealfen^rlnclples with 
actufil apprd^ciliVsV and methods, ^abstyacts of a s^lcjj^tlon of previous ' 
Studies 'of progrW are' In Volume II ^f the publl^tlon. ^ A review .of 

Whe example studies may be useful In approaching one's own program 
evaluation responsibilities, for the abstract^ show how others In t'he 
past have approached and attempted to answer study design ^questions. 
Thus, the user may wish to open Volume/ll to studies of Itnraedlate 
Interest while reviewing principles of jieslgn , formulation presented In 
Volume I, Users should note 'in ^/rtlcular whether the abstracts 
Illustrate educational results, prafctices, long-term results or othfer 
program-related results — such/ as clientele's use- and rating of 

Information received and the sot/rces of such Information, 

/ . ■ ■" ■ ■ ■ ^ ■ 

The publication may bey'read sequentially , I.e., piece by piece. 

Or, users may wish to sklp^/^irectly to Part II and/or Part III In order 

1 ■ f - 

to obtain specific Ideas^/^or designing studies. -Part I is not essential 
for selecting study designs or design facets. Our intention is that the 
publication should s4rve primarily as a resource. . , 

-References tb other sources of published information useful in 
designing studies appear at chapter ends and at the close of Part II. 

Since tWe resource is oriented towards a readership with varying 
degrees of ^expertise in planning and implementing studies of Extension 
program /results, it contains some elementary . material and may 
occasionally seem^ repetitive or redundant to those who are more 
expei^ienced in program evaluation. . ' ^ . . 

' / Whether a study measures immediate program outputs, follow-up 
practices or Impacts, the question of its design is paramount. Desxgn 
is a core consideration in choosing a methodology. Influenced by the 
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^^.ntentionf); \)roci5duree , timing nnd budget for the fltMdy. Only four 

- \ ■ % . ■ ' . 

(Wbigns are /put forward In this resource.'^ They are not tha only. deBigns 

for studies of program results, as is repeatedly stated throughout this 

f ■ • . . ■ \ " 

resource. The designs advanced are not new, but are well-known; indeed, 

• ■ . • ' . \ 

this resource brings them together partly-/^ because they tend to be 
popular among program evaluators, ^ " l\ 

The special contributions of this resource to the subject of study 
designs are: (1) it links overviews of the four designs and alternative 
• facets .within these designs to their application in previous studies of 
Extension program results', as a way of illuminating potential methods as 
"1^11 as pitfalls in future studies, and; (2) it systematically presents 
several Extension programming and evaluation situations which affect the 
choice of study design. Moreover, the publicati(4||^P^resses practical 
as y^jell as ideal scientific considerations In Jthe selection of study 
design. 

Definitions 

The term "program impact" is used variously. Sanders (1982) 

defines program impact as broadly as we define program results. 

Impacts may be intended or unintended, may be positive, negative or 
neutral In* value, may be, stable or unstable and fleeting; may be 
seen at the immed:^ate closing of the program or service and /or may 
be seen a long time following the program or service, may appear 
for primary recipients and/or tertiary recipients j twice removed 
from being directly involved in the program or service. 

Guideline's for the Extension Accountability/Evaluation (A&E). System 

(1983) define "Impact Studies" as technically valid indepth studies to 

assess: (a) the economic or social consequences of Extension; efforts, or 

(b) other aspects of Extension inputs, operations . or programs. Thus, 

6 



the Extension A&E System emphasizes studios of the economic and eoclal 

consequences of Extension programs whl] e allowing^ for other typew of 

« ■ . ■ ^1. ' ■ * . • 

technically valld^ studies as" noted atovet* The ^stresB *bn pconomlc and 
social conaequenccs of Extension programs appeard to be consistent with 
the request of the U.S. Congress (1977) that tfee Secretary of Agricul- 
ture provide an evaluation of these consequences. However, even though 
the emphasis may be on program consequences, the Extension A&E System 
allows for other types of valid studies, such as those mentioned above 
in (b). • 

Accordingly, we define program impact as the ecorfomic, social 
environmental and individual consequences (results) of program-induced 
ieamlng and practices. These ^consequences or end results (Bennett, 
1979 and 1980) emphasize the prevention, checking, reduction or solution 
of problems encountered by clientele. Ideally, evidence of program 
impact ' is expressed in terms of whether desired end results occurred, 
plus detection of any side-effects. However, without assurance that 
clientele have attained a certain level of performance or practice 
through program-induced learning, attributing desired end results to an 
Extension program may be meaningless. ^ 

Because direct measures of program impact are difficult to obtain, 
it has been suggested that utilization (e.g., clientele practices) be 
used as indirect or "proxy" Indicators of impact (Irtioley, et al'. 1970; 
UNESCO, 1979). An impact study could be viewed as a two-way street with 
measurements of participant learning and performance or practice 
pointing toward impacts, and any measurements of end results (the 
impacts themselves)- . pointing back toward the original influences 
(Including- the program) which produced Impact. 



.^^ Id Buwinflry, the EKtenB\lon A&E System InclucTes^ both broad and narrow 
definitl'oQfl. of "Impnct Btu'dy J^^Thc- broa^ . def Initi^^ of impact Btudlea 

includes Indepth studlea of Extenfllon ^inputs, opcraMtnlJ aiid ImpaptR. 

■ » ■ , . . • ^ .... , ^ 

Impact study Inhere capibali'Ssed in'*' reference to the^ full^definitlon 

contained In the guidelines for the Extension A&E System (1983). The 

narrower definition ' of imp^^ studies - studies of the economic and 

social'consequenceB of Extension programs - appears to emphasize the end 

» 

results of the education and practice Induced by Extension ^pro^rams. 

The narrow. definition is a component of the broader definition. 

Our conclusion is this: f'n Impact study should somehow assess a 

program's final consequences: (a) preferably through providing evidence 

i 

bearing directly on the program's end results, or (b) by discussing how 

a program's ' measured educational and/or practice results might be 

* . * 

expected to produce its end results. 

■ ' • Ml^ ' ' 

General Procedures 



The '42 abstracted studies containing findings on Extension program 
results were selected by^ the authors from' a pool of 153 studies 
conducted from X961-1982. This pool v/as comprised of: (a) 148 studies 
of Extension program effectiveness selected^ (in 1978) by two social 
science 're^earoh firms under contract to Extension Service, USDA 



rom 



(Bennett, 1980); and, (b) ^ f ive ad^tlonal studies selected f 

responses to an Exterision Service, USDA, request for 1979-82 state 

■■ ^ ' ■ • 

studies on Extension- program impacts. The USDA co-author (with the 
adviqe and assistance of other evaluation staff members of PDEMS, 
Extension Service, USDA) selected these five studies, using the criteria 



npd procedures adopted' by ,tl\e leml\,flpcial science research firm*. The 
/rosearcK** firms had been' contrncten to Bolect Btudlep with nclcqunte 
•methodological , flubetantiation of ifindipgs and concluelorta voRartfing 

: • • ' ' ; V \« ■ : ^ . , '■• ' • • 

Extension progranv results (Appendix A). • 

Procedures adopted for the .present project Included the following; 



Task I; ' Inventory each of the Vl53 abstracts and classify, the 

studies according to the following four stuyy deolgnsj 
survey, tlme-serle'fe, e^ompailson group, * and field 
experiment, 

' . .-^ ^ * . " 

Task II? Review and appraise each; of the abstracts according to 

the criteria set forth in the Extension Service, USPA, 
request for state studies from the period 1979-82 
(Appendix R) , 

Task III: , Select from within each of the designs several studies 
that exemplify i5everal methods for gathering evidence of 
program results in each Extension program area. 

Task IV: Review existing guidelines and standards for evaluations 

and relate these to studies of Extension program resiijts. 

Task V: Develop guidelines for selecting study designs and 

conducting sj^^udies that examine program results. 

Task VI: Identify facets and- options in Implementing each study 

design, citing examples among the abstracts. 

Task VII: Identify issues in design methodology based on appraisals 

of the methodologies of the A2 examples.* , , 

Task VIII: Draw conclusions and implications for designing Extension 
program results, including conditions under which each 
study design may be selected. 

i-j • ■ ^ 

Task IX: * . Prepare a publication to aid state program leaders and 
^- specialists in their various roles in designing studies 

'of program results, wlth^the assistance of the interstate 
advisory committee and other State and Federal Extension 
personnel. 
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.Tha isi *«tU(llo» '•nVa ^ clnflfllfled omong tha four .fltiidy ^daHlgna /, 



AH followi: 



'I 

.5- 



ff 
I 



Survoy 
/Mme-Hortefl 
Comparison ^roup 
Field experiment 



69 

« 

3t 
11 



/ 



.Hii,ch etudy wan then categorized by program at^Jj^^ 
commiintixt^^^QpTnent/natura!/ resources, 4-H; home^rt^^TO^i^^^ 



coiintj'tlx ten's ion program. The distrlbutloi) of "f^t^p'#,^W;r^^ 
. according to type of design Is presented In Tob.t^- aV;'/^^^^ 

' TABLE r^v A^^ Studios ( f rom WhicWfc^^jfes Were 

* Selected) Grouped by Extevisi^^^^p^tm Areas 

and Study Design ^"^-.^^V'^-pJ^ 

Extensl,on Program kt^ -^' 



5a4 



Study Agrlcul- Com.Dev.& 
Design /ture ^ Nat. Res. 



Survey 

Timg- « 
Series 

Comparison 
Group 

Field 

Experiment 



Total 



?.0 
12 
9 
3 
44 



12 



4-H Home Total 

Youth Economics Program 



23 



11 



11 



17 



17 



30 



5A 



Total 

69 
42 

31 



11 



153 
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Tho l&)%abfltrnctfl were ravlowed by tU« authora «nd « Marylaml St«t*» 
«dvlHory committee of UnlVarpity of MnryXwnd etaff ami St«t« ami, local 
Kxtenolon paraonnul. From theno Abstrftctsi ^^2 ,w;er« selected tci\, 
roprcflent /I variety of methtvlR :f^t obtaining evtdiince on pruBram 
reBultP* Thu ntudlee selected ropreflent botb tbose which required' 
large Htudy rcHourcea and Chose requiring less rcaourcefl. Table II 
preflentB the number of atudien Holected from oach Htudy deaign^accordlnB 
to the Extenaion program areas. It wae aomotlmen difficult to V^^^o a 
given abatract within one des'ign and proprnm area category. 



TAULE TT Abstracted Studies Selected as Examples, Grouped 
by Extenaion Program Areaa and Study Design 

Extension Program Area 



Study 
Design 



Agricul- Com.Dev.A 4-H 
ture Nat. Res. 



Survey 2 

Time- 
Series 5 

Comparison 
* Group 3 

Field 

Experiment 2 



Home Total 
Youth Economics Program 

6 M 



Total 



10 



ToLal 



12 



15 1 



42 



Table III presents the names of the authors of the studies 
selected by study design and by program' areas. , 
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48(aFBtning tha rWulls of K^craH&laB progrdss* Mnwev^r* tht^si^ h,^ 

V 

Ta^ultii, iBfcJttJudulogJiyny ttpe<ikthgt ^ir**ti IUp J'tM»l i»r ffiHs* 

which fhcJ 4^ Weiri «k^|W'iat} diitsb HMl tluiU4*«J all <fheJ Itfed | h^uju 1 i i rt I i 

quant it4tive tituilteci^ n^r fcviewo uf ijua H (oc Ivc ' a(u4ieei pr f ¥t«3a>>l«it^ 
program re^uUti yer^ 4V4il4U\lii, ^l^eoudly, tht: h'} ntUaltcr Wcf^ Icvit**' 

portray ntudl^cl froci tlt£frre\tt' nctthotjologii'dl rtpprfM* , ilinc^imt 
pr*>Ri4m 4rc4ri ami dtftc^rt^iU g^ogrnvhlr nrciiitt of the lnttt*d Strtfi?^, 
Ftnallv, €^Kamni^?^i vt*rr clte^tl whtVh »tghl afft^ct fiuura Hcudtcs ii*Jyin« 
on varylnn levt^ln of rerttMitee avniL-iibtllf y, (Nearly one-half nf th^ 4.* 
rttudl«*t* vrro fund**i^ ^it Ivixnt part 1 ally by F>tt*?rtnlon S<*ivh«^. 
l/T»nA» or prlvnte n^^ccor Hpe^rlnl fuftdt^.) 

• \ Refer CO the Tabic of Contents in VoUm*? 1 1 i^^r i^hHtrnoi^ f>< 
individual titudle« listed *iccordlng CO the four ntudy deulK*^^^* .^tudlen 
nre ordered within a t!c«lpn cateRory by nlphabrrical ortfer of the 
author's narac. Or, individual utudies p.^y br located by con?*uUinf: tlu? 
Index of Studlc?j according to progrAiH atea--where ah?*tr.ncted t;tudien are 
llBtcd alphabetically (by author's name) within program area c.uegorlei*. 
Finally, Volume II Includes an index of studleR by authors' alphabrf t?.ed 
naneB. 

The abstrnrrs included in Volume IT were edited but are not altereTI 
in substance from their courccB In the research contractor publlcat icnr . 
We chose to delete the study apprainalj? by; the res'carch contractors^ • 



roii|Hil«sr ictrJpVat syeira, Appisniii^ C |»rovl*trb ^apcfei^te iHetfUf*' 
tionrt rui ubi^iutng ruil r^porcb of the bfUiUeo of fxttm^ton progfaft 
r#oViVfa vhU'h (^%^ ftittittiUceA in thii» publtccti ton, 



^ifid dccuimfSS^ Ih:? U.?>,uVAt i:*f^u<»l04s ^cfvi^ct, ^ 

Off trr . 

.-^tudicj^ with l!S£act JJtiaif^^ Atlinl^t^m* VA t t'.riJKA. ScUn<<^ 

VaI AiTiht^t . A{»j^Tn4it itrvtil , tt'» ^ur^*u?lHg abiitrdttlo. ^ 

^ P P 5* !; y rt t r ft ft » I n , ( 1 ^ ^ ) , 5 itt^l u ic:^ for t» f rv ^iti^ rmjjg^^t^^^^ 

tfi£^ir Uil , Vo I , HI, Af H nVt^?^^^ VA r Auchur . 

( 1 I ) * ^^2^ ^ ^ o t thf» n4tionai ta«k icrcc on EM <'fj 1^1on accou nt ^biX^^^^ 
awA eVA luAt ion !avgi|g», 

Kv£i I ii.-! 1 1 on 1 ^^^^^^ prc^gra taa: rrc>cr<*i!lnptt f ro» 

alnV-j^r ^^ ryNjluAt Ion prtwork nnnu4il conl igrcner ♦ B.t It ! r*c^r«* • Kt>.* 
October cnnpiUd by hf . K. SoUh (ppflT-l^)* Calnc^vlllr, 

Fl.: Cooperative Fxtrn^lon Service, ^ 

Scl«»nce Hanagectent Ccrporattcn. (1979). £nvento^^ 

of rtfportR\abnCr4ncted , Vol> 1. V^ishlnf^tDn. 1>€: Author. 
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CHAPTER 2 : THE PLACE OF THIS RESOURCE IN EXTENSION EVAtUATlON 

• ' . ■ . * • 

Traditional models of programming tend to portray evaluation as one 
pha^e of . a cyclical process — the phase' following program planning and 
program implementation. ^^^^^ 

However, recent models suggest that evaluation is a part of each 
phase of programming. Different 'authg^rs divide the overall programining 
process into different phases, or title ,the phases differently.^ How- 
ever, all of the models deal with the questions af deciding what kind of 
program to have,, then how to conduct it, and finally deciding on 

• ^: ' ^ - - 

.improvements. , 

For example. Stuff lebeam (1971) Identifies evaluations which help 



to:^ 



• Select program purposes regarding; int^endedy clientele benefits 
(evaluations of program context ) ' 

* - . .' " 

• Select program designs capable of achieving the desired 
clientele benefits (evaluations of program input s) 

• Improve program designs or their -implementation (evaluations 
of program process ) - 

• Improve program designs or purposes jbri order to better achieve 
% intended clientele benefits (evaluations of program products ) . 

Some authors subdivide the four dimensions referred to by 
Stuff lebeam, or attach different names to these dimensions. FjOr ex- 
ample, input evaluations may be called formative evaluation; process 
evaluations^ may be called program monitoring. Relatedly, Dave (1980) 
divides evaluations of program products (r^isults) in^o those which 



examine shortrterm prop.ram results and those which examine longer-tern 
resultR. 

Our resource publication focuses on studies which help to evaluate 
program products, outcomes or^esults.^ Three levels of program results 
are defined and Incorporated in this resource: 



Educational results : these Include • changes In clientele 
knowledge, attitu^s, skills and aspirations. 

Practice . regiults : these include clientele patterns of 
behavior, ac^fcions or performance gemming from educational 
results . 

End results : these ' incl^ude the consequences or impacts of 
progirhm-induced educational and/or practice results* . 



Figure 2.1 shows that these three ^eveis of program results are 
levels 5, 6, and 7 in a levels of evidence model (Bennett , 1979). 



Figure 2.1 ^ 



7. End Results - 
6. Practice Change 
5. KASA' Change (knowledge , attitudes , skills, aspirations) 



4. Reactions * 
3, People Involvement 
2. Activities 
1. Inputs 
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••Value of Information" studies also receive attention In this 
resource. These studies are closely related to studies of Extension 
program results, and ere based ' on participants' ratings of the value of 
./Information they received from Extension, and ^from other sources for 
decisionmaking In some field of endeavor. Such evidence may imply the 
existence of program results although the nature of such results is not 
specified. 

Our resource publication provides an overview of four ma*n study 
. .. ^ ... ^ , ■ 

schema for selec^Hfe,- collecting and using evidence In evaJtiatlng a. 
/ ' , f 

•program; I.e., four study designs, j These designs are: survey, 

time-series, comparison group and field experiment. 



Evaluation Users j 

The question of who will use the evaluations of program results is 
central. Schmidt (1977) dlstlngillshes .between policymaker users and 
program manager (leader) users. The ^rlous types of program results 
are differentially germane to dif f|erentll^ind of users. Different kinds 
of users may use information on particular levels of program results, 

and not use information on other levels because of their differing 

\ ! 

responsibilities. j 

To illustrate this point,' Rivera (19^2) constructs a pyramid 

composed of program evaluation users from the top down (Figure 2.2): 

• - Policymakers (e.g. legislVtor-fej state and federal 

executives) who oversj^ pejLlcy^^^^ 




Policy adminlstratojis-^or thepprogram (e.g. Extension 
directors) 

Program managers (e.g. , Extension program leaders) 
Program staff (e.g.. Extension specialists and agents). 



19 
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Figure 1.1 
A Pyramid of Evaluation Users 




As a general rule, the higher the position of users In the pyramid, 
the greater their need for Information on a program's end results 
(Impacts). Users high In the pyramid have less need for information on 
practice results and even less need for Information on educational 

results. • 

Conversely, the lower the position of users In the. pyramid, the 
greater their need for Information on educational and practice results. 
Users lower In the . pyramid have less Immediate need for Information on 
end results. 

Generally speaking, policymakers and policy administrators are more 
In need of evidence regarding end results because this Is more useful 
for making policy decisions on whether to initiate and/or continue a 
given program, and how many resources to commit to it. Program leaders- 
and program staff are more in need of evidence on educational and 
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practice results because these ore more useful for making judgments 
regarding the extent to which a program is exhibiting effective and 
efficient progress toward reaching' the desired, end results. It should 
be clarified that our resource publication focuses on a variety of ways 
to obtain evidence on program results, and we only touch here on the 
extremely important topic of uses of .infdrmation regardi;ig program 
results. 

In^ considering different designs for identifying program results, 
it is important to select or construct a design that will provide 
intended study users with evidence of program results in which they are 
most ^ interested. - Of the A2 abstracted previous studies containing 
Extension program results, 19 include evidence on end results (as 
contrasted with educational and practice results). 

Steps in Evaluation 

The abstracts in Volume J.I illustrate different facets of the four 
study designs. These facets are related to^he identification of a 
series of steps in carrying out a study of program results. The 
literature on evaluation is . replete with steps considered essential In 
identifying program results. 

Knowles J(1970) conceives of four steps in evaluation, to: 
(a) formulate the questions to be answered, (b) collect . the data that 
will help to answer the questions, (c) analyze the data and interpret 
them in relation to the questions asked, and (d) propose modifications, 
of the plans, operations, and programs in, light of the findings. 

The United States General Accounting Office's Assessing Social 
' program Impact Evaluation; A Checklist Approach (1978), provides a 

^ "21 



Byotcmatlc frnwowork for organlKlng ovldenco on program remilca. It 
highlighter the procedures of: (1) evaluation planning i (2) data collec- 
tion, (3) data analyala, (A) reporting findings and (5) data dlscloaure. 
It placcfl cmphaaiB on the political nature of evaluation, and reminds ub 
that evaluation is not Juut for program development and improvement but 
may alao be for policy Justification or change. 

Bedcr (1979) formulates 'a more elaborate planning and 
implepientatlori strategy bf ^ight steps to successful evaluations, as 
follows:. ■ ■ .' J 

1. Decide on the purpose and use of the evaluation. 

2. Determine what will be evaluated. 

3. Acquire and allocate evaluation resources. . 

^ 4.,. -^Establish. a. propeir climate (participation and cooperation). 

,. • 

5. Choose an evaluation design,, or approach. 

* ■' * ' 

6. Con4uct the evaluation. 

7. Report the evaluation. 

8. Act on the evaluation. 

Cunningham (1980) identifies six stages of evaluation, beginning 

with establishing a steering or advisory committee (a stage consistent 

with the modis operandi of Cooperative Extension) . These stages are: 
* ■ 

1. Establish -a Steering (or Advisory) Committee 

2. Determine Strategy . 

— Purpose and Use ^ 

— What to Evaluate 

— Resources and Budget 

3. Agree on Design/Approach 
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A. Conducr/Implement 

5, Report Reflultp 

6. Act on ResuUa 

A Hteerlng (plnnnlng) or ndvlaory (con^uXtatlvti) committee c^n bo 
valuable In providing advice and nlao "conaent." A conoonaus of opinion 
on a moot point may prove of considerable value when Jufltifying certain 
plans and actions • 

Kappa Systems i Inc. (1979) complfeted a three-volume review^ 
appraioal and summarization of studies of Extension program 
effectiveness, including guidelines for improving evaluations of 
Extension programs. Limitations in study methodology and reporting 
common to many of the studies appraised were identified and used in 
developing ten guidelines for future ntudiea. These guidelines, closely 
related to steps in evaluation, are listed below: ^ 

1. Clearly state study purposes. 

2. Specify study limitations and/or degree of generalizability . 

3. Describe the Extension program being assessed. 

A. Relate study questions and measures to program objectives. 

5. biscuss the reliability and validity of the measures selected. 

6. Establish a link between client outcomes and Extension program 
delivery. 

7. Provide adequate labelling of tables, charts, and graphs. 

8. Separate presentation of findings from conclusionG. 

9. Provide adequate support for conclusions and a comparison if 
program success or failure is concluded. 

10. Balance . completeness of report with succinctness of 
presentation. 



The focua of thin publication la on Beder'a step number five abovn, 
and on Cunningham' a atapa number three. The puldellnft In Kappa System'fl 
U«t which relates moaft, closely -to formulatlnR study dealgn la that of ^ 
No. 6: "Katabllah ft link between client outcemea and Extension program 
delivery." A, central queatlon lii the aolectlon of a atudy design .la 



prc'claely thl 



la: Whnt logic and facta wlU be adduced to determine If 
and t« what extent there la a link between client behavior or atatuo and 
I the delivery of an Extension proRram? 

Kappa Syateraa alno dlatlnguleheB between Btudy flndlnga on program 
reeultB (guideline 8) and conclusions about program aucceaa or failure 
•(guideline 9). Conclusions on progi^am success require well-formulated 
criteria, generally based on program objectives on other expectationa 
for the program. ^ V < 

-Procednre.B for formulating and using criteria in evaluating the 

success or failure of Extension programs (Steele, 1970) are beyond the 
intended sbope of this resource publication. Hence its title refers to 
studying program results rather than evaluating program results. 
However, systematic eval/uation of program .results is usually impossible 
without sound evidencd on the nature and extent of such results, 
.scertained program - Jesuits should . lead to answering evaluative 
questions such as, "old the results justify the amount of resources 
invested?" "How badly were these results needed?" "Are other programs 
which receive similar kmounts^ of resources accomplishing more?" "Should 
program staff try to increase the magnitude of intended results of J:his 
program in the futurer" 




In summary, our 
one of the steps 



publication on designing studies focuses on only 
r stages In planning, conducting and utilizing 
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0/«lu«t;i«n». Ab neceaBary, re«dera should consult other references 
which deal with the other eteps In evaluation. The inteUlgent 
selection or conatruction of study design depende upon adequate comple- 
tion of the.Btepfl preceding It. Moreover, uaing study findings intelli- 
gently (in evaluation, accountability, and program management and pol'icy 
doclfllons) depondfl upon understanding the nature and limitations of the 
study design which produced tho findings. * 

St andnrda for Pvaluntiong 

Two major 80tn of atandarde for evaluations hav^ appeared recently: 
♦ ■ • ,- * 

a) The Joint Committee on Standards for Educntlonal Evaluation. 
Standards f or EvaluatlonH of Educational Pro;^r<^m8t Projccts» 
and Materials ,. (198177 * 

b) Evaluation Research Society Standards Committee. "Evaluation 
Research Society Standards for Program Evaluation." (1982). 

The Joint Committee's Standards arc organlred under four nxaln 
headings— utility standards, feasibility standards, propriety standards 
and accuracy standards— and among the 32 standards Included, some 15 
refer to designing evaluation studies. One of the strengths of the 
Joint Committee's stait^^ards Is not equating high quality program 'evalua- 
tion with technically accurate study methodology. For example, how an 
evaluation study may be used can be more Important In appraising* Its 
overall quality than Its technical accuracy. Technical fl^^uracy Is 
necessary but does not Insure appropriate utility. 

Of particular Interest is the section of the Evaluation Research 
Society's standards on structure and design. This section Includes the 
statements? 

The ^design for any evaluation cannot be conceived in a vacuum. 

It is necessarily Influenced by. logistical, ethical, political, and 
fiscal concerns and therefore must take these as well as 
methodological requirements into account. 



— ror all types of evaluatlona, a clear approach or deatgn ahoulij 
t)t» specif te»1 and Justified aa appropriate to tlie types of 
conclualone and Inferences to be drawn. ' ^ ^ 

—For impact atudlea, the central evaluation dealgn problem of 
isfltiroatlnB the effects of nontreatroent (absence of program) and the 
choice of particular method for accompli ahlng this ehould he fully 
deserthed and justified. 

Prior to the publication of the two above aeta of standards, Kappa 
Syftema, Inc. (1979), under contract^o RxtenalonvServlce. USPA, estab- 
Uflhed atandarda regarding technical accuracy} I.e.. regarding ^bstan- 
tlatton of flndlngn nnd conclualona In studies of the «f fecttveneep of 
Exreniilon programa (Appendix B). The A2 studlaH of Ext«n»lon^)rogr«m 
results which nre abstracted end analysed relative to facep of atudy 
dtislgn all meet the ©tandardn which Kappa Systems adopted ir^scon^ulta- 
rion with the Science Management Corporation (1979), 

Thus, while the accuracy standards employed in thia^ retiource 
publication arc only a subset of the standards relevant to approlnlng 
studies of Extension program results, they are a fundamental subset. We 
assume that a study pf program results can not have real utility unless 
it is 8uJyi9ilently accurate. Our publication does touch upon considera- 
tions of propriety and utility^ and emphasizes the criterion of feaslbil- 
ity in the selection of study design. 

Conclusion 

With this b'rief review of uses, users,, steps and standar<|s. In 
evaluation, let us move to an examination of study designs. As stated, 
the several alternative designs presented are not new to the literature 
on evaluation; thus our' purpose is to bring these designs under close 
focus, systematically identify design facets and illustrate their value 
in formulating studies of Extension program results. 
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Su-^^^h iMti^fAts rr-Milta itKlMtic* r^tnf i^rc*^iHl5* kf»A^vit?4i|c5 , 

rttututlrt. jikilU ar d^plraUr-nl* <KASA t linage) ; imilvMuAl or f^olUiftvci 

I m _r 

U la Itsportant ttirtt ^lii-rnatc' (rlvrtl) «• »p 1 nn<.l Ions of these chAnpt-n 
.1 JfrJnntea or <akcn Inta rtt count . Tlu- lienJgn of « »:tuijy t« thi* Mjor 
frtctor tn iJMf iVnlnlnji; how weU U r.nn nttrll.ut.' clleul.'l*' behavMt>jal or 
Ht.iuin ch.inftA-s to r.xt«?n?ttou as corapnr«-il to non-Extcn«>on Influcnceii. 
Studv drfiign« vary In th»^lr .-iMHty to account for > nlt<»rniite 
e:<pliinnt lonji. 

Study defilnnn KcncrftUy mnWe or Inply nome fom of comp/irlfion 
^cither within or between proupn. When a vithin^grotip HeslRn Is u?icd» it 
nav or ray not be knovr bcforehnnd juf^t what <!1 f f cront peoples expor»ure 
to /I program has been or the Intensity of their program tpartlcipatlon. 
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chiingrn in r^i^eLitiir^ bcfhavlorn or nfniu^c^^ pr^gr^?^ par { i*^ t » Caa.:h 
4a chati^,€' in rt^tu-frl t d|u») With h.-W ofrc'^ Ot<*;^* par H ^' ij^at e . Cctief- 
411^, vlthin-gruup ijr?iljj,tu: proviac uuly dt f^i^n amt toX ovct . 

rival r^f^Unrtt ii n?i but tnrtic jt^l t«Tjii r^-ly on &c^chv4r^ of rtt^it.i#t 

wtrh the prcj'rnra. 

Stronger i^vlclenrt* thnt rllmlrJr betuiviiMrtT or «tiiluri eb.mgeti itre. 
In fnct, beCAUfie of pnntclpnrlnK In n progr^n (^tn oppn«ca tu rival 
expliinntlon?0 My be provlJed when cor^pnr 1 nons nrc r-iidi? btftwrrn tv^ or 
rK)re groupn (Campbell and Stnnlry . l^bb) . Tbe^^e bi^twcrn-group designs 
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compare specified charncteristics of program participants with those of 

a comparison or .control group ^ith 1) no participation or, a varied 

degree of participation; or , 2) participation in some alternate program* 

' Such comparisons can Help assess the role of the Extension program in 

f 

influencing participants apart from non-Extension influences. If the 
program group and a compari^son group are subject to the same npnprogram 
influences, any differences in relevant changes between the two groups 

can be attributed to the program. Two difficulties exist in using 

t , ' ' 

between-group study designs: (a) finding a non-program group, coniparable 

to the program group, and (b) dealing with the general expectation that 
>:all members of the target program audience will have the same opportun- 
,.ity to participate in the program (MAGI, 1979). 

The following description • reviews . four selected, study, designs 

il V . ■ 

■ commonly used in evaluations of Extension program results.* A simple 
means of describing ■ these study designs is by way of the matrix in 
Exhibit 3.1. The h orlz on ta^l labels of this matrix refer to the type of 
comparison planned-^-within a group or between groups. The vertical 

' labels refer to >7hether' the design requires pretest or benchmark data. 



Exhibit 3.1. 
Matrix of Quantitative Study Designs 



DESIGN REQUIRES 
PRETEST 0R» 

RFMrHMARK" nATA ^ 


WITHIN-GROUP 
DESIGNS 


BETWEEN-GROUP 
J DESIGNS 
* 


No 


Survey* 


Field Experiment 


Yes 


Time-Series 


' < * * 

^ Comparison Group 



* Survey, is uspd in the same context as in BennettVs (1979) description 
of study 'Resigns. It is not to be confused with the use of the 
word "survey" to mean simply a -misans of collecting data. The term 
is used both ways, in the literature, but for bur purposes it refers 
to a "onte-shot" design for studying participants (and sometimes 
others) only after the fact of program participation. 

» • " ■ » 

The two, within-group designs included in this publication are the 

survey and the time-series . More formal names for these designs are, 

respectively, the "single group posttest only" design, and * tKfe single 

group pretest/posttest" design (Frefnch, 1983). 

The two "between-group" designs most commonly used to evaluate' 

Extension programs are the' ^comparison group and the field experiment . 

■, ■ • * . ■ ' " 

More formal namesV for these designs are, respectively, "quasi- 

' ' --4 < . 

experimental pretest/posttest" design and "true experimental" design' 
(Frenth, 1983). , 

e 
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The Survey Design n 

When It is decided to ascertain results of an on-going Extension 
program (one already being Implemented) and there afe.no benchmark data 
available, It Is generally neceissary to Implement a survey design. A 
survey design may complete a study of a program's results more quickly 
than the program that produced these results. This Is because, In the 
survey design, data f rom cllentel e are collected only once; e.g., 
following program participation or at the conclusion o^ the program. 
There are two major types of surveys used in the evaluation of program 
results: perceptual and cross-sectional. 
, Perceptual Surveys ' 

Perceptual surveys generally show program participants' perceptions 
of a program's results. Such perceptions are based on participants' 
reflections or retrospections regarding their behavior or statuses 
before the program,, and their estimates of the change in these behaviors 
or. statuses . because of their participation. This perceived 
"before-to-after" evidence is one way deal with the question of 
attribution, i.e., what influences created specific behaviors or 
statuses of participants? ^ 

Surveys nay also obtain * perceptions of - program results from key 
observers of a program who do not participate. Observers provide their 
perceptions, based on retrospection, of the results of a j^rogram for 
participants and other people to whom the participants relate. 

Some stilidy users question the validity of findings on program 
results based tipon perceptions of clientele as to Extension's influence 
on their behavior or statuses. House (1980) distinguishes between 
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"subjectivist" and "objectlvlst" posltlonB or assumptlon» about the 
nature, methods and limits of program evaluation, Objectlvlsts consider 
thfit .evaluation Information should be ••scientifically objectiveT/^ 
achieved through reproducible, quantitative techniques be verified by 
logical Inspection. Those who maintain an objectlvlst point of view 
would argue that reflective or retrospective data does not produce 
adequate evidence of program results. Objectlvlsts question the valid- 
ity of retrospective data , asserting that what clientele perceive or 
believe about the results of a program may be Influenced by what they 
want to ba^^leve. Furthermore, retrospective data may be Invalidated by 
memory loss or distortion (Rossi and Urlght, 1977). 

Those who maintain a •'subjectlvlst" point of view base their claim 
for validity of perceptual data more on an appeal to experience than to 
••scientific objectivity. •• Subjectivlsts hold that human Experience is 
perception — that it is necessary to. discover what program participa- 
tion means to clientele, and to allow them to connect events Iti 
identifying the. cumulative effects of multiple experiences in program 
participation. It is also advanced that perceptuai^f Indlngs on program 
results are easier for most study users to understand and use than are 
quantitative documentation of program results (Forest and Marshall, 
1977). /. 

An important use of the survey design is to gather perceptions or 
opinions about the activities and results of Extension programs. In 
many instances, representative sap^ples of program participants • and key 
observers^ perceptions of Extension program results may be 'evidence 
sufficient to meet identified study users^ needs in. evaluation and 
decisionmaking. 
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opinions are surveyed In may areas> such as: (a) the extent to 
which Extension program objectives have been achieved; (b) the extent to 
which Extension »and other actors, agencies, etc, have produced given 
clientele behaviors;, and (c) how satisfied Extensldn cllcntele are with 
the results of the programs. 

Cross-scctlonal Surveys 

The cross-sect Ibnal type of survey Is more consistent with "objec- 
tlvlst" approaches to studying program results.. Compared with perccp- 
tual surveys, cross-sectional surveys rely less on program participants* 
perceptions and interpretations, and more upon logically and 
empirically-based quantitative analysis. The same may be said fgr the 
time-series comparison group and field experimental designs* 

The cross-sectional survey generally includes participants of a 
program and nonparticipants as respondents. Such cross-sectional 
surveys include all those ~ ox a sample of those — who are in a 
demographically or geographically defined population (e. g. , all farmers 
or all adults in a state). Such a demographically or geographically 
defined population wlM include both program participants and 
nonparticipants or, the cross-sectional survey may be confined^ to only, 
program clientele, making sure to Include tbo 5^1^ with different kinds or 
degrees of program involvement for comparative purposes. 

Cross-sectional surveys tend to obtain data on: (a) respondents' 
self-reported behavior, qualities, or statuses at the time of the data 
collection (and, perhaps, at an earlier time) ; (b) factual, situational 
data such as ^ self-reported participation In Extension and other pro- 
grams^ age, formal education and family status, etc.;. and (c) data based 
on clientele records. 

■ i ■ ■ ' • 
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Two possible variations within the croee-sectlonal survey doRign 
phould be mentioned. 

(1) • Data collection only from previously known program partici- 

pants. Here, the cross-sectional survey can only compare 
clientele with' different kinds and degrees of program 
Involvement. 

(2) Data collection from previously known progi^am participants and 
a random ^^Pl-® °^ g;enerai populatl'on In order to Identify 
and obtain responses from nonpartlclpants. This approach Is a 
substitute for sampling d* defined population and still gather 
responses from prog;ram participants and nonpartlclpants. (For 

. , other variations In survey design, see Clark, 1976). 

To deal with the question of how much an Extension program may have 
contributed to specified clientele behaviors or statuses, data from 
cross-sectional surveys are subjected to multl-varlate, statistical'^ 
analyses, such as: successive statistical adjustments (Rosisl and 
Freeman, 1982), factor analysis, discriminant analysis, multiple 
correlation and regression and path analysis (Costner, 1971; Mosteller 
and Tukey, 1977) . Such statistical controls are necessary because of 
the Inevitable lack of equivalence In relevant characteristics among 
program participants and nbn-partlclpants which may affect specified 
behaviors or statuses of clientele. It must be determined to what 
extent such specified behavior or statuses are aAually attrlbuta)?le to 
the program rather than to nonprogram factors. 

Appropriate multl-varlate analysis of survey data can help deter^ 
mine the conditions which link program partldlpatlon to results. For 
example, program participation may be associated with clientele practice 
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change only among thoHO cllentole having oovtain personnUty Craltp 
(o,R.*, Innovntivonesa) or aoclb-economlc atntuHefi. NonproRram factora 
can abBcurc or oxnggorate Htudy flndlnRB on program reaulta, evqn with 
roultl-vnrlnte nnalysOB, 

Voluntary participation in Extenaion programB 1b a nonprogram 
factor which Is difficult to control statifttically • For example, even 
with complex multl-varlate analysis in a survey. It in difficult to tell 
whether: (a) program partlcipantfi who are more In contact with an 
Extension program therefore use more recommended practlcea. or, 
(b) program participants who use more recommended practices have sought 
more frequent contact with Extension and other sources of similar 
information, or, (^/a combination of (a) and (b) are actually the case 
over time In some cy^ellcal fashion. ' ^ 

While multl-varlate analysis may provide much statistical control 
over nonprogram factors which confound study f lndlngs\regardlng program 
results, findings from such multl-varlate analyses still may be mislead- 
ing. Multl-varlate analyses rely upon many assumptions .about the nature 
of quantitative variables In the analyses and the typical relationships i. 
among these variables. It Is often difficult to meet these assumptions, 
or to know the extent to which they arc net: falling to meet such 
/assumptions can substantially affect study , findings . The factor pf 
probable differential motivation by program and non-program individuals 
may be addressed by modeling the process of self-selection into E}^ten- 
sion program participation. However, this technique has so far had,' 
limited acceptance (Rossi and Freeman, 1982). Thus, authors of survey 
studies are advised to take great care in (or avoid) inferring that an 
Extension program is a causal tf actor in measured clientele outcomes. 
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The T lme«"8e r t o R Pe a 1 gn N 

* 

When the doclHlon has been made to Htvuly reflultH of an onjtoltip; 
program, and' benchmark data on pruBtam partlctpanta are available and 
applldable to the program results to bo examined, additional data may bo 
collected in order to make tlme-oeriea comparisons. Additional data 
collected on partlclpantfl during or following the program arc compared 
with the benchmark data on the aame individual particlpanto. ) 

Time-series designs are used roore frequently in Extension ^ to 
examine repulta of a new program (implementation will begin in the 
future) or, an ongoing program, with. a new set of participants. Here, a 
pretest Is administered before the program begins, and these pretest 
data on clientele characteristics arc compared with posttest data on 
these same individuals. 

Time-series studies which .collect data only twice (e.g., before and^ 
after program implementation) should be distinguished from time-trend 
studies which repeatedly measure 9llentele behaviot or statuses as they 
relate to program objectives. An important technique in time-trend 
.studies is to compare time-trend projections of pre-program data with 
observations of the participants, after program implementation. An 
indicator of program results is the difference between the observed 
••after" specified behaviors or statuses and those projected based on 
past overall time-trend data (Hatry, et al., 1981; Rossi and Freeman, 
1982). 

In time-series studies, comparisons with nonparticipants are not 
made, but comparisons may be made between the extent of "before-to- 
after" change among participants with varying types or degrees . of 

• • ■ , 
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participation In the program. Tiin^i-^arlea dr^igna «r0 not «« efface J 
no betwean-Brpups ijealRns when Imllc^cing how much chanRo conaiatent 
with proRrnm ob.U»ctlvoa U duo to participation in th<? program or to 
nonproRnim InfluenceH. Reporta on tlmu-aerlea atudlea ahould acknow- 
ledgti and addr<?B« tho poaaiblo Influencii of appUcabl« non-program 
factors (e.g., fluctuation in weather conditions on irultl-year changeii 
In crop yleldfl). However, tlrae-Beriea atudles may provide valid eatl- 
matop of program reaulta where other po^elblo aourrcB of change in 
clientele chnrncteristlcH cnn be ruled out logically. Also, if Revoral 
tlmc-Gorles fltudlcs of a given program arc carried out In multiple 

r 

BlLcB, each producing similar fttudy findings, rival explanations become 
less of a thrc^4[^^^Tlme--8erlefl studies often attempt to establish 
statdfitlcal controls in order to determine If or the extent to which 
participation In an Extension program leads to specified behaviors or 
statuses. • , * 

Program participants' experience In responding to the pretest may 
Indeed be an Important stimulus toward measured change In their 
behaviors or qualities which are sought by the program. This pretesting 
experience may serve as a stimulus which operates in addition to. the 
Involvement In program activities. Special design techniques may be 
employed to address the possible Influence of the pretest. 

The Comparison Group Design 

When the decision has been made to study results of a new program 
or of an ongoing program which will have new sets of participants, the 
comparison group design may' be employed. The comparison group design 



atteroptH to eatabllBh high degree of 9lmUarily between program 
PMrttclpantfl and a group oi noiipttrtlclpanta. Pre-program inforraatlon 
about rhoae who will be exposed to the program and thosP who will not 
rtllowH the conHtructlon of nlratlar groups. fju/h/'cunatructwU controla" 
(RophJ «nd Freeman, 1982) may be accompliehejl by matching, as much an 
poflflible: (a) chftraccerleticw of Individual bairn of participants and 
nonpnrtlcipantB; (b) atat l Htical dintr 1 but 1 o rf» of f.allenc cbarncter- 
latlCH of nets of partlcipanta and non-partic\lpants, or; fc) In tact 
proupH of partlclpnfita and non-particlpnnts. \ 

The comparison group design is limited since pJhtching of program 

i * 

I 

participants and nonpartlclpantfl la on]y partial and not complete. 
Thus. Rtatifltlcal control-a are usually added to a degree of constructed 
control. In using thiR design, Llie constructed cciitrol and statistical 
control are combined in order to attempt to remove or reduce tllfferences 
in the characteristics of participants and non-participants which might 
flffej^t the clientele behaviors or statuses which are sought by t^e 
program. 

Authors who use a comparison group design p,re generally advised to 
address^ explicitly the degree of comparability of the two groups con- 
cerning characteristics which might pose an alternate explanation to 
that of program effect. This meansS:clentif ying factors in addition to 
the program which may effect the changes specified, so that at least 
these factors may be accounted for , statistically in assessing Exten- 
sion's degree of contribution to program objectives.^ Statistical 
comparisons of, or controls for, program and comparison groups include 
characteristics such as age, sex, socioT-economic status, and. aptitudes. 
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Statlptleal techniques auch co=«varlance an«lyKl« or muUiplti 
ragr*iBflJon can correct partt^Uy for auch ext;raneou» factarw. The 
program luul coinprtriepn groups m^y b# compared etatletlcaUy In term^ of 
before«to-i«fter gain acore» Char are adjueted to account for lnttl4l 
lUfferencf^a in "before ncorea.*' However, It may he difficult to com- 
pennat^ for nny lUffercncea In motivation which led program participanta 
Into involvement with Extenalun (Alexander* 1965). Thin rtelf-t^<?lect ion 
blaH m/^y not be a problem in Mome comparlaon group atudlea, PoHHlhle 
cttloetH of the pteteat or the poatteat acorea of i\w program and eompar- 
Lho\ ;;^oiipji f hould nl«o bo conHldered. 

The Field Kxpertment Dcatgn 

When the deci«ion has been made to study results of a new program 
or of an ongoing program which will have new sets of part lcipant8» the 
field experiment design may be employed. This design Is also known as 
the "randomized group design" (Smith, 1980). The field experiment Is so 
called because it was Invented for agronomic field research rather than 
for biological laboratories where constructed control predominates. 

The field experiment requires making the program available to 
clientele selected randomly (through chance alone) from some potential 
audience. The part of the audience receiving no exposure to the prognun 
Is the "control group," I.e., the Rroup which does not participate In 
the program or participates In a different program. ^ ' 

In the comparison group design, the set of Extension program 
participants may "volunteer" to participate in the 'program, while the 
set of non-participants chooses not to be in the program. This key 

41 51 



differiiici^ in thti v5!>iiinUria» of the two aeta — partlcipai^ ^mj 

non-part l«ip«nta m«y poti^ ^ nertous thra^it to tUt^ viilliUty iif-t^tudy 

ftmitnga on progrtiw r^ttuUx. Ab expUined abpva, thii«e who **Kpr*iri?i {h§ 

iluBiftt to p4rftt:ip4U In th0 program mi^y U4va a 8rt?ttt«ir tnitlwl moti- 

k 

vtttion to iicliiev*j the program'^ ohj^kctlvaa th<*n tho^e who 4a not 
volunit^t^r. 2 

But» in th^ ^Kpt^riment , tt 1» poaaibU to ranUowly assign 

perflooH who have alr^^nly voluntetired for program p^rttclpatlon to: 
(a) participate in the Rxtt?n8lon program ("treatmtint;*') » or (b) aerva in 
a control group. In thi« ca«0, rival explanatlona for clientele 
behavior or atatun can be more completely accounted for than in the 
prevloualy diacuHHed dealgna. 

In utlllxlng random aasignment* the program and control groupn are 
not expected to be initially equivalent; i.e.. all nonprogram factora 
are not constant between the program and control groups. Instead, the 
field experiment design randomizes uncontrolled variablea among program 
and control groups (Fisher, 1935). With random assignment, tests for 
statistical significance can be employed in order to determine the odds 
that any greater increase in program group achievement over that of the 
control group was brought about by the presence of the program rather 
than by uncontrolled factors or chance. 

In one type of field experiment .observations within the program 
and control groups are made both before and after the program. The 
types of observations InN^uch a field experiment relative to levels of 
program objectives and clientele outcomes have been depicted graphically 
(Bennett, 1979). In a second type of field experiment, observations 



within %\w profir*i» €Q«t*ol groy|>^ mix MSBl nm^^ 

' , i 

«: 

eoncrul group, tttsmb^ns. AUhaugh Uie rt^l4 ^}«perl»#nt my be che mueit 
*»ff<5ectv«? tl^tijgn fur eeii^nttf tc^Uy eoimd coneiypltmii^ r^g^rding progrdm 
reauUb, It le often dUficuU U) r^ndottly <i8»tgn p^rttclp^nlH In «n 
Kxt«nflton tiduc4tion4l atJttlng. Thoaa who dtsfitrid to parclcipAte in 
KKtifOQlon pri>grd»a gen^raUy expect; i^^ml tre«t«w^nt» Ponying p«irtlci« 
padon In whflf f^eemn to be n d«?itlrnbl«? progrd»# or of fearing vh^t 
to bu a than Idertl program to certain participantii Buiy po«i» uchtcal 

qu0fltlona, Howaver. it may be raplUd that until a atudy of program 
rosttltfl haa bi?ro conduct<?d. It l*i not known whether ih^ program ia in 
r<»allty of benefit to clientele (Bennett and leonard. 1970). One 
possible compromise may be to delay offering the program to the control 
group until after the experimental group has completed the program. A 
second appronch Is to asfilgn volunteers to different variations 
of program intensity, without having a non-program control group. 

Summary Compari^^on of Designs 

While the matrix of Exhibit 3.1 provides a simple way for comparing 
and contrasting the study designs, certain dimensions of each may exist 
within the other designs. The survey as a technique may actually be a 
component of all th^ other designs (Clark, 1976). For example, the 
time-series design involves collection of survey data from participants 
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gMbibit 1,2? piausHiiJontt of Four Study ptesiigfia 



Study 
Daa tgn 



Dura CeU*setf?4 Tr^c^nt or ^ Conp^rlaoj* 



at Conclueil^rt 
of Program 



Survry 



Comparison 
Croup 

Field 

Kxp<?r laent 
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chapt^;r Aj options in designing a study 

Selecting A Study Design 

The study designs discussed above offer varying degrees of 
scientific evidence of program results regarding the extent to which 
KASA changes, practices, or end results were effected by Extension 
rather than other sources. However, selecting' a study design Is a 
thoughtful process that cannot be technically or mechanically 
standardized. There are advantages and disadvantages In each design, 
and what is gained in one may be lost in another. 

While many evaluators regard the field experiment as the ideal 
design study for measuring p^ogratp results, Cronbach (1982) defends 
non-experimental designs for program evaluation. He differentiates 
between the job of the basic researcher and the evaluator, pointing out 
that advocates of experimental research design often assume that fixed 
and limited questions will be answered through successive studies over a 
long period of time. Study users, on the other hand, often call for the 
illumination of a whole program and its results within a comparatively 
brief period of time. Gaining a strong cause-and-ef f ect answer to some 
of the many questions about a program's results through an experimental 
design must be weighed against the cost of leaving unanswered many other 
questions about the ptfogram's effectiveness. Thus, Cronbach suggests 
that selection , of design for a study may be based on whether to 
sacrifice breadth of Information, that could be gained through a survey, 
for the sake of more definitive answers to narrower questions. • More- 



over, the added power of an/ experiment to attribute observed clientele 
changes to a program also raises^ the risk that the study design will. 
Interfere with convenient program Implementation. 

Compared to other designs, the survey generally requires fewer 
resources for any given scope of programming evaluated • The survey 
design may be selected In cases where it would be difficult to justify a 
study requiring a greater amount of time, money, or -other resources, 
which would be perhaps required by the other designs^ The survey may be 
selected when the geographic or programmatic scope of the study is 

extremely wide, thus requiring considerable resources for even the 

... ■ ' . t 

survey design. 

A well-designed survey can provide substantial evidence regarding 
the exte;^ of a program's results. However, .assuming that other aspects 
of a study are of equal quality, the survey is considered by many design 
experts to provide -weaker evidence than between-grpup designs on whether 
specified clientele changes are attributable to the pi^ogram in question 
(e.g., Campbell and Stanley, 1966). As mentioned earlier in this 
chap/er, the survey design relies either upon respondents perceptions 



program results, or upon statistlcjil controls to achieve evidence of 
a program^s contribution to specified clientele changes. A well-planned 
and .executed survey can be more valid than a poorly executed 

b^tween-group design. • -v- ^ * 

Cronbach (1982) asserts that: "For any evaluation many good designs 
can be proposed, but" no perfect ones;" and "designing an evaluation is 
an art." There is not only a choice of designs, but numerous choices 

in. 

within each design, and those choices will and should vary from sltua- 
tlon to situation. While it is the study designer's task to produce the 



moBt useful evidence, he or she Is working with a specified amount 
of resources - financial resources, staff resources and respondents' 
(e.R. program participants' ) cooperation. Consideration must be given 
to these factors and several other facets of the study situation when 
designing a study, of program results. Moreover, the study design must 
be chosen with -a view-toward how the study will be used in mind. Thus, 
the utility, feasibility and propriety of a design must be considered in 
addition to the design's capacity for providing strength of evidence on 
program results (Joint Conunittee " on ^ Standards for Educational 
Evaluation,. 1981). * 

You, the reader (as the organizational manager of an evaluation, an 
evaluator, or advisor to an evaluator) may select or help select " ideas 
from the designs described and the examples provided in this 
publication. When you begin to choose or plan a design, remember there 
is no one best design for all situations. The choice will depend upon 
the interplay of several variables, including information needs of 
identified study ■ users, time constraints, resources available, 
creativity, personal/ judgements, anticipated criticisms, information 
available and political climate. There will be no automatically correct 
or incorrect' choice of design. Instead, the thoice must be a 
thoughtful, creative process taking into account the realities of the 
Extension programming and study situations, and balancing the various 
trade-offs. Recall -too, that while we have presented the study designs 
as individual models, a particular study may require a combination of 
designs. , 

In addition to the' following overview of facets of study designs, 
suggestions on design selection may be found in Chapter 6 of this 
volume. 




Overvlev of Facoto of Study Design 



Ten facets of study design which are addressed below may assist you 
in choosing or formulating a study approach. The facets ennumerate 
i[^yarlous considerations In choosing and executing a study design. While 
presented as separate considerations, the facets are Interrelated and 
must be considered In light of each other. We have not attempted to be 
exhaustive In our coverage of design facets: other facets not treated 
here are methods of reporting to Intended study ^.users, and ethical 
considerations. 

The facets of study design which we address are: 

1. Program situation 

2. Type of study objective 

3. -iKferources 




'5. 

6. Data sources 

7. Sampling strategies 

8. Data collection 

9. Nature of data 
10. Analysis of data. 

We wlllV discuss the facets above In the following pages with 
emphasis upon choosing or determining a specific option or options 
within each facet. Then7 in ChagS^^^ 5, some of these variWs options 
will be illustrated, for each of the four study, designs by citing 
previous studies of Extension program results abstracted in Volume II. 

: 50 ,58 : : 



We hnve proHontod the following flection aa though renders will be 
determining, fllonc or as a group member, which of the options for f^udy 
methodology to select. It la . assumed that program lenders ond 
speclnllsts will be nble t^ identify withHthis perspective and apply it 
to their roles in planning studies, e.g., manager for Extension spon- 
sored program evaluation or member of an advising or steering committee 
for an evaluation. , ♦ 

The key to t^clecting the appropriate optiohs for designing a study 
is two fold: (a) ascertaining who needs what information about program 
results for what purpoBfes\ arid (b) fitting \this "demnnd" for information 



with the options (c; 



'suplply" of such information which are actually 

feasible. The question, '^'why is the study of program results needed?" 

y ■ > } / 

must be reconciled with/£he answer to, "What kind of study is feasible?" 

■ // 
The ten facets /^fe presented in separate figures (4.1-4.10) and are 

analytically brolJ^n down into options to help you visualize the choices 

to be made i^'tVi^egard to each facet; each illustration is then followed 

by varlqiis "considerations" that spell out the visual analysis.- 



4 
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Figure A,l: The Fflcot of Program Situation 



ProRram aituatlon 




Statun of the 
prograro 



Primary source 
of program 
direction 



/ 




Ongoing with continuing 
participanta 

Ongoing with new 
participants 

New program 




To^ determine the situation regarding the program selected for 
Evaluation consider: ^ , 

1, What is the status of the program? 

Ongoing already with clientele who are. continuing to 
participate? 

Ongoing already, but to be conducted with a new set of 
participants? 

A new program to be Inltated with an audience who will 
begin to participate in the program? 

2, What is the primary source of program direction? ^ 

The federal level of Extension (because the program is 
federally funded)? 

The state level (because the state or campus funds the 
program)? 

The local level (because of county or city resources 
supporting the program)? 
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riKuru ThQ Facot ol' Type of Study Objective^ 

,K(liicnt lonal 
.prnctlcefl 
> ImpnctH 



Type of 
«tudy 
objective 




Defirrlbr ptoKiam; 

IHViUltB 




■^Del Ivery- modes 

DcRCvlbo .program 

Tctailcs by progvnn Staff expertise 

variables 

\ Participant 
characteristics 

Describe program 
results relative 
to program costs 



Value of program 
information to 
clientele 



Individual 
Economic 
Social 

EiWlronmental 



To choose a type of study objective, consider whether the intert is to: 

1. Describe how the program par ticipants or others changed or 
benefitted as a result of the program? 

Wiat have, they learned? 

What practice* have they adopted, changed or continued? 
What are the individual,^ economic, social and/or 
environmental consequences? 

2. Describe the results of a program relative to its variables? 

Is the delivery ntode related to results? 

Is degree of staff expertise, or some other attribute, 
related to results? 

Are characteristics of participants related to program 
results? ~ 

3. Det^mine the least expensive way to achieve intended program 
results or determine extent of results t hat can be obtain ed 
for a given cost. 

A. Determine the value perceived by clientele of the information 
supplied by Extension and by other sources for their decision- 
making. 
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Flgura 4.3; Th^ Facet of Rt?6outce« for th© Study 



Time 

Exptirtia© 

To determine the rcHOurccs available consider: 

1. Which Exten«ion staff are available to conduct the study? 

How many <hourG are available from Extenelon staff? 
What types of expertlso are available? 

2. Is volunteer help available? 

3. Will funds be available in order to retain non-Extension staff 
to conduct or help conduct the study? How much money is 
available? 

..»■■■ ^ . 

4. Wliat types of university, Cooperative Extension Service or 
academic department support will be provided? 
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Flgurt^ A, A; The F^cut of Tt»ia Frames 



Time 




•During proRr^m 
•rpst program 



Time orientation^— Immediate 



Proftpcctlve- 



. Short 



. Projected 
Actualized 



To dotcrmlne the time fraraen for the Btudy, consider: 

1. At what polnt(B) in time will the datn be collected? 
* » . 

Prior to the prograni? 
during the program? 
After the program? 

2. Wiat will be the time orientationCs) relative to program 
results? 

Examination of past KASA change, practices or end 
results? 

From a program conducted recently? 

From a program conducted a longer time ago? 

Examination of the Immediate results of a current 
program? 

Examination of longer terra, results from a current or 
future program? 

Collection of data now on clientele' s intended 
behavior? 

Collection of data at a later point in time on 
clientele's actual behavior? 



ERLC 
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Flgurt* ^*t5i Tht* Factor of Study Scope 



.Olographic 




Contt^ntj 



Single county 
Mult t-coui^y 
Stilts 



Slnfil» program 

aren 
Muitl-proRram 

area 

► Ov<*fall program 



One flub-area 
Several eub'-ar^ao 
Entire area 



Audience 




Specific 



General 



determine the scope of the study, consider: 

The geographic area to \)e covered by the study? 

A single county or unit within the cnnnty? 
Mor^ than one county? If so - which onep? 
N The entire state? : 
More than one state? 

Which picogram aren(s) will be examined? 



One program area? 

A single sub-area? 

Several sub-areas? . 

The entire program area? 

More than one program area? 
The overall Extension program? 

Whether the audience population will be the specific target 
audience(s) of the program or a more general audience which 
Includes the program's audience. 
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rtgura 4*6 J The facut of Mt^ Sowrcae 



! 




observcra 



To determine the datn sources for evidences on program reaults,. 
conttldcr : 

1. Which audiences will be asked to supply data? 

Progrnm participonts and/or comparison or control group 
persons? 

Persons having indirect contact with the progrnm through 
the program participants? 

^ 

The potential but yet unreached program audience? 

2. Will data be squght from the program providers? 

3. Will data be gathered from key persons and/or community . 
leaders who observe a. program but not necessarily ss 
clientele. 

■ y . . • • 





tiit'lvr 



Sir 





Hf^npouHt? r^lr 

Purt iclpau^! attrition 



Quota 



Propartlon^it?* 

t»y i»({§t^ 
Dii^proport ik'oate 
by ©trattt 



To choorte a HCratofty for f^rlocclng a snmpU of program provldorri 
rtnd/or audience, confiidcr: 



1. 
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4. 



Whether to studv tHe entire population? 

Whether to select a representative fiample of the population? 

Ratulomly from the entire population? 
By quota from the entire population? 
Randomly from stratified sections of the population? 

With stratified arcatj represented proportionately? 

With stratified area*; represeatcd dlsproportioi^tcly 

for the sake of sampling efficiency? 

Whether to select a purposive sample, where the goal is to 
gain exploratory knowledge rather than to generalize? 

Demographic-based selection of sampling units, in order 
to assure examination of varying conditions. 
•'Snowball" or other reputatlonal ly-based methods of 
sample selection? \ 

How large a sample size Is required to permit the precision 
and analyses desired? » 

Considering the expected response rate? 

Considering reductions In numbers of participants by the 

end of program implementation?^ 






K^^ards or 4ocufbeHiek 




To determine how to collect th* il4t«»» con^ldi^r: 

i. How Mny tine* to culUct dnta frbn ciich inf ort^ittlon c^urc*?. 
Only once? 
More than once? 
What trchntque will b*^ uaed to collt^ct dntrt? 

Interview prnj^rnns providers nnd/or pnrt i<"lpflnt«? 

In p«?rHon? - 
By phone? 

In group fieflalonn? 
Directly ivbfierve beh.ivior within thr ntudy nxitv^pie? 
Dlscrlbuti* a questionnaire? 

Fxnmlne personal or public records ^^nd documcntP? 
Who will collect the data? 
, Paid progran personnel? 
Volunteer program personnel? 
Nonprogram personnel? 



Figure 4*9: The F.acet; of Nature.^of Data 



Data 




Self-feported 



Test scoresa 




>Direct iStservation: 



^Ppljiiions on current conditions 
Retrospective perceptions 
^Perceiyed amount of change 

over time ^ 
-Factual situation or status 
Record-keeping 

.Paper-and-pencil 

-Psychomotor , ' 



— S^tudy personnel 
-Third parties 



To determine the nature of the data to be collected and analyzed, 
consider: v . ! 

1. Will the data be "self-reported? . \ 

^ - Data on respondents*. percept*tons or opinions regarding 

their current iehavi^r,' status or surroundings? 

Data on recollections (retrospections) of behavior, V 
status or conditions? 

Data on changes in behavior, status or conditions based 
, ^ on comparing- retmspective' perceptions and current 

4 ■ percept tonsv. 



Data-^iin fa(>li 
or age) . 



iective) or status^ (e.g. occupation 



Diata 
ob 



2. 




ta generated froni records based on -i:<espondenti5\,.^^^ 
cfeTvatiori^ < 

prograij^prq^ders and/or audience be tested for 



or si 



"r&ough:" pafer-ari<fe^en tesitis?^^^^. , ^ ; 
through psychgiubtor skill examwiations? 



3. 



Will the da^lon ^^irograin providers and^r audience be . 
generated mm dlr^^c^ observatic*ns of behavior or behavior^ 
traced or ^^^^ da^.ta sources? ^\ 

Dat^^^sed observations by stu4y Bpi^sonnel? 

Da^aMased^ ^1^^^^ parties' (e.gt family member^';, 
. coinl^ilki observations or 'records? ^^^^^^ 




Figure 4,10: TheFacct of Analysis of Data 
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^ - ^ ' «"s 

Confidence Intervals 



Hypothesis testing/ / 
tests of significance 



.To choose methods for analyzing data to ascertain program results, 
consider: . ■ v 

1. What types of descriptive statistics, will be used? 

Will there beaveraging generalizations about clientele 
. outcome"s? 



2. 
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Will 'variability among participants be examined with ' 
regard to clientele outcomes?, 
, -$ ■ ' - ' ■ - ' ' . • ^ - 

Will associations between clientele outcomes and specific 
pr^ram and/or audience variable's be examined? 

• Will there be an examination of the simultaneous relation 
cyf two or more program and/or audience variables to 
/clientele outcomias, i»e, the use of statistical 
controls? ^ 

■_ , . . 

What types of inferential statistics will be useii? 

' • ■ " ^'^i ' ' / " ' ■' ' •' ' 

Are there plans for providing "best estimates" of 

clientele outcomes among the total audience for the; 

program, based oh findings from the study sample? 

Are there plans to provide probability margins'for best 
estimates of clientele outcomes among the total program 
audience, based on sample findings? 

Are there plans to test hypotheses that stated clientele 
outcomes have occurred among, the total prbgram audience, 
based pn sample findings? • 



The foregoing facets, and options within these facets, have been 
dlscuRsed by many authors In the fields of educational and social 
research and public program evaluation Including Rossi and Freeman 
(1982); Smith (1980); and Weiss (1972) y Our Intent has been to^complle 
and graphically present a type of "checklist" to aid study design 
formulation within a whollstlc picture of the study planning process. 

The following chapter will offer a more detailed explanation of 
each design as it relates to the facets outlined above. 
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CHAPTER 5: STUDY DESIGNS, FACETS AND EXAMPLES 

In this chapter we analyze each of the four study designs according 
to facets described in the previous chapter. We cite specific studies 
abi5tracted In Volume II in order to exemplify some of the options within 
the facets. Chapter 5 may serve to familiarize readers with the range 

0 

of example studies in Volume II. . . 

Survey Designs 

Types of Study Objectives 

Survey studies describe program results in terms of different 
levels of evidence. Studies abstracted in Volume IT exemplify program 
'participants' opinions as to the extent of their knowledge change 
(Steele and Everson, '78), attitude change (Forest, '77b), increase in 
skills (Tait, '69; Rockwell et al.,.'82), or aspirations (Rockwell et 
al., '82), as a result of program participation. Perceived practice 
change also is exkmined using thi^s^esign (Goetting, '82; Williams , ' 78; 
and others). End results examined by a survey include perceived econ- 
omic benefits (Glass and Reese, '76), perceived improvements in health 
and saf^ (Porest- '77),' and increased self-confidence (Rockwell et 
al., '82; Steele and Everson, '78). ^ 

Surveys' may describe program results according to such factors as 
presence or degree of clientele participation in an Extension prograift 
(Brown, !467, Dardeau, / 77) . ' ^ 



■ ' " . • * 

Survey atudloa mny nlno. ahow clleivfccle'ff perception of the role of 
KxtetiHion in providing Bpecifled types of Information, This typo of 
Hurvcy generally g'nthern ,dnta from n repreabntntlve nample of a defined 
population and may be calledj, a 'Value of information flur>Ajy'' (Awn and 
Crowderi '77; Bogenschneider i 



rf '77). 



Time Frames 



Surveys lend themselves to the retrospective examination of program 
results—short term (Kanarek, '78; Talt, '69) and long term (Rockwell ^ 
al,, '82). Some surveys* include prospectivl data on the actions of 
clientele, e.g. stated intentions or - aspirations to engage ,ln specific 

■ • ' '^ " " 

practices which are recommended by an Extension program (Goetting, '82j.' 
Scope of Studies 

Some of the survey examples Included In Volume IT examine a program 
In only one geographic unit or county (Awa and Crowder, '77), others 
exemplify nultl-county surveys (Rockwell et al., '82) and others 
statewide surveys (Goetting, '82; Steele and Everson, '78). 

While the majority of the jsurvey deslgnvexamples In Volume II 
focus oni a single subject matter area, Rockwell et al. ('82) and Summers 
* and Zeller ('78) examine more than one subject matter area; Forest and 
Marshall ('77) cross program areas and examine results of tfhe overall 
Extension- program In one Wisconsin county as perceived by participants. 

data Sources 

Besides collecting evidence from program participants, surveys may 
also gather data from third parties, for example, parents of 4-H part'lc- 
Ipants (Steele and Everson, '78; ' Sumnfers and Zeller, '77). Data^ 
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collection from tbo potential Audience for a progrnm 1h exemplified by 
Awn nnd Crowdor ('77) and BoRenschnelder ('77) • Dnrdeou ('77) and 
ForoHt nnd Marolmll ('77) nlflo domonatrate collection of data from a 
random aamplo of n geographically defined population! to dlacover both 
the total extent of partltlpati on in a program and the self -reported 
results of participation in tha^ program. 

Data may be collected from community leaders as observei^s of 
program results (Forest et nl., '76). Data may be obtained from public 
records as a supplement to survey data from clientele (Frye and Miller, 
•76). . 

*■ ' 

Sampling Strategies 

Studying a- small number of participants. Forest ('77b) and Class 
and Reese ('76) included the entire population in surveys*. Simple 
random samples were used by Bogenschneider ('77) and Kanarek ('78). 
Stratified sampling was used by White and Ladewig ('79) and Tait ('69). 
Williams ('78) selected sample size expressly to be lar^e enough to 
insure a sufficiently accurate representation of the population. ^ 

Data Collection . . / 

Techniques for data collection in the survey include the face-to- 
face personal interview (Dardeau, '77), the telephone interview (White 
and Ladewig, '79), mail questionnaires (Glass and Reese, '76), or some 
combj^nation of the above (Steele and Everson, ' 78) . Brown ('67) 
demonstrates how survey data may be supplemented with data from direct 
observations made by interviewers. Forast and Marshall ('77) demon- 
strate the use of non-program personnel as data collectors. . 
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Nature of Data 

i 

Solf-repo'rtod opinions on reBpondentn' current farm and woodland 
production practlcua wore colloctod by Brown ('67) and Dardeau ('71). 
MoHt of the flurvey exampluB In Volumo**IT nre baaed on/'aplf-reporta of 
p^-ogram participants regarding the Amount of perceived cKm^^^ their 
behavior or change In qualities due to their Kxtensiqn program partici- 
pation (e.g., Suipmers and Zeller, '77). Wlilte and Ladcwlg ('79) ellcl- 
tated rcfipondents' (a) retrospective perjceptlons regarding their past 

behavior, and (b) perceptions of their current behavior, comparing 

""" "' " ■ p 

self-reported behavior before and after participation statistically. 
Analysis of Data 

Most of the survey examples In Volume Tj[ utilize percentage distri- 
butions of self-reported pro-am results . for participants as a whole. 
However, some of the examples compare the distribution of responses of 
program participants exposed to different program delivery modes (White 
and\adewlg, '79). Length of program participation Is correlated with 
•the extent of self-perceived program results (Rockwell et al., '82). 
Dardeau ('77) examines associations between participants' characterise 
tics and adoption of specified practices. 

Summary Statement on The Survey Design 

The survey is a valuable design for conducting studies of perceived 
program results. If nonparticipants are included in a survey, a variety 
of multi-variate statistical analyses can be employed to learn Exten- 
sion's contribution to clientele behavior or status. Surveys can be 
comparatively simple, inexpensive and less time' consuming to implement 
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,t\\t\\\ eh^ other tli^HlRnn described In t\\U\ veaourca. Aim), the Hurv<?y Ih 
flexible and can he uaed to 8t?cure t^vldenci? of reaultH of i program that 
iH alnMuly^ In pro^^rtJHH. 

Volume n contatnn abatractH of 17 atudlea which utllii?;©d « »urvoy^ 
dcHlRO. (Six of thu 17 Incltldc findlnga on «nd reauXta, J au program 
impactH.) Five additional Burvey attidleR -not abstracted In JJ^lumo II 
are referenced in the «ectlon on Further ReadipRa-Survf y , wl^ch appearH 
at the end of this chapter, 



me-Scrles bcsigns 



Types of Study Oblectives ^ 
Change in clientele knowledge and attitudes has been measured using 
pre- and posttcsting (Crowe et al. , [76). Kingdon and Toensmeyer (*75) 
compared knowledge change as well as practice change at six-month 
intervals using self-reports of .food consumption to determine a pro^ 
gram's point of diminishing returns. Time-series designs have docu- 
mented program end results in terms of economic gain (Strickland et al. , 
'76), increased agricultural production (Ladewlg and Edmondson, '72) and 
positive self-concept (Marks, '71). 

Time-series designs have also been employed to examine program 

results ap related' to various delivery niodes (Texas Agricultural Exte^- 

... 

slon Service ,'74), program intensity (Carter, '80) and degrees of staff 
expertise (Marks, '71). Marks ('71) also examined program results in 
relation to specified audience characteristics. . 
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In rtdtlitiow to coll<ic?tlnB data both buforii nnd ftftur progragi 
piirtlciput Ion (mvrtnmu uml Urown, •70), ac^me^ Ume-seri^n Htudlt*« collt^ct 
<|rttn for a third time (parhapn months «fter t\w clofia of participation), 
in order to rheck for maintenance of ai^ cllenttUe progreMS (MarkfJ, 
Other tlme-Herlea Htudtoft collect data at lnt*?rvalfi throughout 
proRram participation (KlnRdon and Toenflmeyer, '75)t Some tlme-flcrieH 
Mtudien obtain data on program del Ivery or program participation over 
the duration of a time period of program implementation (Carter. 1980). 



Scope of Studies 




Time-aerie« fltudlefi can be carried out within a single geographic 
unit (Crowe et aU, '73) or in mMlt;^.-Unlta (Ladewig and Edmondaon, '72)* 
The time-series design is particularly applicable when studying a single 
subject over a wide geographic area (Carter , '80) . since limiting the 
program subject matter helps to simplify the complexities involved in 
multiple data collections. 

■ ^ ■ ■ • : ■ / 

Data Sources 

' ■ - — * , 

In addition to collo^Tlng data directly from the program partici- 
pants* supplementary ptjrceptual data and opinions may be collected 
through surveys o^^^^^LOimunlty observers (Crowe et al., '75; Balliet, 
'78). Supplementary survey data may also be collected from those who 
are in turn "clientele" of the program participants (Brown and Nelson, 
^Nj77). 

* 
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Kingdom md Tnenameyer ('75) eampled famill«» from ««ch of t]\r^^ 
rouiitltirt pruport lonrttiily to ch§ number of program furalUeM In, thesi? 
refip^ctlvi? t^ountiiJH, Jn *» prnportlonftl rftudom aumpl^. Cart^sr CflO) 
emplbyetl a jiC^itowlde random 8flropl€5 of dairymen In n survey conhtned wJth 
0 C Jfne-ser hfB ritudy. 

■f 

Dat a Colle ction 

Hro- and poat-queationnaireH an employed by Crowe et nl, ('75) are 

typical of a data collection technique commonly used In tlme-«erlo« 

studlcfl. Records kept by clientele over a period of time may bo 

employed to study economic Impacts (BalUet '78) and changes In agrl- 

cuttutal production (Ladewlg and Edmondson, '72), Pre- and post- 

program data gathered In personal Inr&rvlcvs can provide evidence on 

' ♦ 

practice change. Ladewlg and Edmondson ('72) collected post-program 
only data on participants' perceptions of change in their status tn 
order to supplement a time-trend analysis. - Likewise, a mail (question- 
nrire sent out by the Texas , Agricultural Txtension Service ('74) pro- 
vided auxiliary survey data to the time-series analysis of the study. 

If data on Extension program staff or clientele can be obtained 
from an existing database rather than through a special data collection 
effort, the task of data collection may be simplified. For example. 
Carter ('80) exploited state-wide data, on Extension program activities, 
by county units, collected routinely during a period of time through the 
Tennessee Extension management information system. 
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The miijorUy of tliB t Irae-tterluM RtudleH preisenteil tn Vohim** U 
rX|iroaa program reaultH percentage 'ch«ng«s. Hflrka ('71) »«««^ 

rtiirtlvrtlfl of varum-.? to teHt whether acorett lmUc»tlitH Htilf~eBt««m 
jm-reaHcd nlRolf teantly (Ht«CiHt icaUy) from the protest Co th.> 
purttt«>HtH. ami wheth^ir thero wah a .llffer^nce if\ program rp«uUrj by 
profeaalonfll «nd non-prof eafllonftl Kxtenslon Strtff, 

In tIftermlnlnR tht-' rel*it lonahlp between program rtsaulta «nd other 
varlableB, llnrtman nnd thrown ('70) used regrrHulon «nalyflla to determine 
whether proximity of Kxtenslon demonstration fnrmfl predicts changeQin 
farmers' practlccH. MultliJle regruBrtlon annlyala employed by Crowe t)i 
al. ('7'>) determined the relationBhlp between program resulta and 
frequency of pnrticlpantB' attendance at Fxtension meeclnga. Carter 
('80) Invcfttlgated the percent of variance In dairyiticn'a acoeptflnco of 
recommended dairy production practices that could be explained by. extent 
of contact with Extenfjion through applying linear and curvilinear 
regression analyses. 

Summary Statement on the Time-Series Design 

The time-serien design may be selected to obt^^iX^J^nior measure- 
ments concerning changes in behavior, qualities or status of program 
participants than data based on perceived changes. Compared with the 
surveyj a longer time period is generally required for defining program 
results. The design may be suitable for examining program results when 
using a, comparison group is impractical or unnecessary. 

Ten abstracted studies included in Volume II use the time-series 
design for assessing program results. (Six of these 10 studies Include 



finatnga on program iJfui r^ibuUti or Imp^tttt,) Thti r^^it^r my also wiah 
tt) r**rtir to four c iwa-derittti Htuiltea whlrli 4r*3 lUH <ibfltr«freil tn 
Vnlumo 11 bur hv^ rareiisnc**cl in tlit* Further ReatlJn^a rtt ch^ (^ip*?^ of 

t 

Tvpea o f JU udv OH j^c t tvt^a 

The eomparlBon prmip ditnign h*iH been \i*eit to compare program 
pnrtlclpnntH ^^nd non-pnrt IclpAntn with resp^'ct to change In knowlK^df.o 
rsubalma. '61). nttitiidea (Street, '73), and aiS^Kr^tlone (Green t?C ill., 
•Xn. Practice otiange ban been f*imllnrlv examlrtd (Aldrlch. WO. Fnd 
renultM miMuuired in compnrl«6n group di&»iBnn have Included "quality of 
life" (Street *73), labor efficiency (Alexander and Longest ^ And 
Incoir.e (Aldrich, '75)^/ Comparinons of program re^ultK havt be^in made 
regarding part i^cipants who have received: different program activltlea, 
I.e.. treatments (Mauk, * 7S) ; program de 1 Ivery by Btaff wUb varying 
levehi of expertiue (Street, '73); or. InstructtonaT activitien In 
different learning environments (Boone and White, '76), 



Time Frames 



\ 



PoattCRting in comparison group studlea can occur immediately 
following a short term wCrpjCram (Boone and White. •76). a longer-term 
program (Aldrlch. '75). at set intervals throughout the progrnn (Green 
et al.. '72) or at some interval of time following program participation 
(Subalma. '61; Marks. '71). • 
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ii ' ' 

t'owpdllbUU Br»'MH jcbltins hav^ hpen used {U c><!irnJH«e thts IBoUllb «f ^\ 

«LMi «ri*up eitutJtdt^ roiu^f on one dulijci t within ri iMOgfrtfi^ Ahlrit'li 

♦ 

lUta toilet te.l dtrp' tly. frota proBi-ntti « ^ ^"'^ 
p.trclctprtnt'* mny br »iippUtaent«-ii with obHurvat lona by thtrd party 
obifrviTH fSubalma. '<in. Host comparison (troupH rc>pr«8«ntfd fl»onft the 
rttudlPK w«'r.. h-iti^A im th^-lt objccUva alnilnrliy to th^ r.^^p^nlv,* 
prop.cnm Rroupf. of the Hludlrn (e.R., Alixandt'r nnd l»np.c<Ht, '(>?'', Hnuk 
•7"»; Street, '73). In cuntraat. Grocn el nl. ('12) ^fltnbHohcd 
rewparlson Rmupn conpriH«^ii o! doHlr.n/ited friends of mcnbern of the 

* 

- Sam pling StrategU's • ' * ^ 

A census of pArticipantJi and comparifum Indlvlviunls In uf^cful when,! 
the studv concentrates on a program of Hmlced size and/or snail geo- 
; graphic area (Mauk. '75). Raml^J sampling within stratlMi^d sections of 
a gencrnl popMlatlon wan employed by Crycphagen ('69). while Alexander, 
and. longest ('62) used a modified rnndon sample of program participants 
and non-participants," Street ('73),' employod stages of sanpring with a 
convenience sample followed by a Btr_ati.fied sampHnp procedure."* 
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the d&ta . can Incliijde: 



personal 



. i W'^"'" i'^^)?^^^^ techrftques 6ol}.ectlng 
■rl/' ■ 'in^ ^the telephone lnteVvlavf^|^^^^^^^ 

'"^^ documents' (A^^y^^er and llongest., ' 



White, '76). While program personnel mny 



, ?x^g)^^:^t^ ^oll®9ti"8 dlata» GruenhagenJ'^^i9) and 



-Mv program. 

- ■ ' • Xi kiii'-aMf^^^l^P^ and Longest 's ( ' 62) study . was 

S'^^S^ sM^ofc''t^^^ while, Sul?alma (,'61) supplemented 

'^.^^^ite,itV^^ teachers' perceptions of these youth . ' Relf- 

fv5' •:^'^epoi"te global in nature ^Street, ' 

or practice (Gruenhagen, '69) . 



youth. 
73) or specific to a 



* ''^^^^^ .program and comparison groups often differ onv'pretest 

^ \Rcares, the groups may be compared statisjtically in terms of gain .-scores; 
adiusted to account for initial difference55 (Boone and WhiteJ^;^' '76) . 

Analvsis^bf ebvariance with statistical tests may ci6|termine th^v.sigirdfi- 

■If'..- v''^ ^ ■ ' ' V " ■ -V' .■•••'y ^f'o 

cance of the differences between 6r among thfe; grou|>^:^?|pruenba^ 
/ \. Subaiina\ '61; Alexander and Longest?, '62) .. . - J^^^'V^ " v ■ ; . 



Summary Statement on the Comparison Gfoup Design 

The comparison group design can '^Ip to' possible 

of 



.influence on clientele outcomes of factors ; other; than^vthe ,^ 



program, ^ Ilowevet, . there may be -problems In galnlng:![^h6'''Cooperatlgn. of 
nonpartlclpnntB and In achieving sufficient, afctiia'l ' or analytic cbmpar- 
nbtflty botween program and comparlson^^groupp J ^ .Such prob,lems make the 
''.^.v comparison groijp design more difficult to ira^j^yl^r^ than the twp within-. 

group designs, . Insufficient comparability mav4 lead to erroneous concdu-^i 

• ■ ■ - • . ■ -v- . , 

sions regarding program results, V^^'/v -'-^ 

Eight studies are abstracted in > Volume. II/ which employed a 



comparison group design, (Six of the eight . studies InclufflF^lndings -^dn . 
program impacts orv;^rid reg^^ An addit'lonal^^six; s^jt^^^^l^^^ 



the Further Readings , 



F le 1 d Exp er imeti t T!) e,s i gn^ > ^ 



^ Typos of/ Study Objectiveis ' ; . 



^' The fi^ld experiment is. effective in denidhst.tat^ng Extension 
^prpg?cam^ practice -change , . as all^ of . the . Volume II 

abitractR illustrating the field' experiment design show, Bowering. et 



al, ( V8);^t end results of ^a nutrition program with- -a field 



experimental deWgn. CompaYisons of, progS^^jm re&ults in relation to type 
. V^of delivery mode were made by Ilonnold et al ; ( ' 8lX> tjrent arid Kinlaw 
('76) also used the field experiment to compete delivery mode effective- 
ne^ss as did Bowering . et al.. ('78). Honnold et al. ('81) uniquely 
compared program results to Extension cost per program participant . 
■ Program results c^n also be examined in r^elation to ^participant charac- 
1 = terlstics (Trent and Glass, - ' 77; .Scherer , ' 77) , 



-1 = 
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Time FrauieB 



^■^^^^ It 1b possible In one variation of the field experiment to collect 



datfv regarding :program participants only at the conclusion at the 
program. However, pretest scores, or benchmark data, can facilitate 
field experimental analyses by providing evidence on degree of initial 

equivalence of the participant dnd non-participant gA)ups (Trent and 

< ■ 

Glass, '77) . . - ^ " ' . } - 

>Som^times pretesting may not be desirable du,e td its "reactive- 
effect. " That is, studies of pretested indityiduals indicate 'that^h^^ 
tend to improve their peribrmanc^ on the posttest regardless of their 
type of prograiSS; eHppsure, or even with no exposure to the 'program (Trent * 
and Kihlaw, '76). • The pretest Itself may actually conceal the program 



(treatment) effect by sensitizing members of the control group so that 

- ■ . . y " " . f ' ' . ■ 

their posttest scores are affected* Therefore, the pretesp^is often not 
Mmlnlstered whetj^ .cond^cting^a^ field experiment (Scherer, '77). 

Scope of Studies ' / 

-The field experiment design can be applied to studies of anjr 
geographic scope . Bowerlng et a/] . (^' 78y studied clieijtele of a single 
b^by clinic, while oth^'r'ileld^^^g^^^ have examined jprogram results 

In several countles;;;|^<e^^^ and Glass, '77), in sample counties qf the 
entire state (Trent and Kinlaw, ^ and across two states (Honnold et .v- 
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al., '81). 



Sampling Strategies . 
Field experiments that are narrow l.t\ gebgraphlcal. sbope may st^^^ 
the entire program population (Bowe^JuigTet al. , .',78) , while those that 



^ arc? broad Ip gcogr'apl^cdi^^^^^^ of parHgipants 'Tiiny« *v4 

. • ' ' ^ . ■ ; ■ - . . 

' use filmplc rartdom «ampllng (Schorer , '77.) or a two-atep/.ranfpm aampllnR 

procesB (Trent, nnd: Glass,' '77). Oh ,6ho other hand, Ibnho] d et ;a,K ( fS^jl^f- • 



conducted a censuB of participaiitfl even though the program hnd over 



participants enrolled, ? . ; ^, " - 

Data Collection ' ' 

' ' Afl demonstrated in five of the fiel^ experimental studies - in 

.. Volupie^II,r t^e personal interview can lend itself to this desigrtf. Paper 

77) as well as a mailed 



J^': y<X (Trent and Glass, ' 

' ' V " ,dve B t ionna i re (llonno Id e t a 1 . , '81). 
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Nature of the* Data 



The fl^d experiments abstracted in Volume 11 illustrate the tise'of 
tests to miEja^ure knowledge change (Honnol'd^^ et al, , '81) , ; ' sc^dM 
'./Jmeasur^attitude change (Trent and Glass, 1977) ^and^tructured ^^f- 
' ..repop^^ practice change (Yerka, '7A). AnjexceptiOn to the 

tendwdy for field experiments to use highly quantified data is the 
. examipation of participants' and nonparticipant controls' perceived 
' practice ' fehange^i^^ V66) . '^^^B^^ 



Analysis of Data 

As shown in all of the Volume II atstrficts, some form of statisti- 
cal analysis compares differences in the mean test scores of .|hG program 
a^d the control groups/ THls analysis determines if these tiiffefences 
are statistically significant. Several studies also examine the 
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bfi'eitttion betwoe in participant bdTidvlor and /charncter- 

iHtlcB: (o) pf the ExtonBion program provldera (Yerka, ' 74) / and; (b) of 
Jt he program piw&tlclpnntfi. (Trent and GlaflSt '77; Scherer, '77). ^ . 
, ., ' ■ ; . ■ ■ ' ■ ■ ' ■ 

• qymmary Statement on the Field Experiment Design ' 
' The field experiment can provide strong evidence of an Extension 

program's results by Recounting for other factors which m^y have 
affected clientele. However, the field experiment Is more ef fect4*ie'' In . 
Studying structured, narrower-scope programs than programs with a wider 
scope in content or \eub'Ject 'matter and with less structure, ''Use of the 
field experiment requires clqse coopetatlon between program managers >and 
progtam evaluatorSV in order to satisfy study, design requirements, 
V 'Whe^. It ' is feasible and ethical to use random assignment, field ^ 
• experl^iental findings can b^ very scientifically persuasive in^meas^iring 
*Extensl6n program results. . . 

« ■■ "J-:,- ■ - • ■■■■ : ■ 

Volume II co'ht^ins'^abHtracts of seven studies using a f leld ^experi- 
inent/desSgn. '(O^ry 'onie ofy the seven sltud^ies. includes^' findings on 
; /program ^impacts or end resu^ Kn additional study is referenced in 

the Eiirther Readings/ : i » ' " / 

General Comments and a Renyincfer . 

The citations 'above, ^ to studies 'in Volume II which exemplify 
^ options within the. design .facet^|^are not ati . all- exhaustive r m more 
options within the facets/could be exemplJ-fied by tTie studies. We hop^ 
-\ tha;: a sufficient^'nutnber of facet options haye. been explored to ^suggest 
^ the richness of th<5 Volume II abstracts as ;a resource for planning 

future studies of Extension program results. ' . :J 

.... •'• w •'>••.'..■*'.',*■ , ' ^ • ^ ■ 
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AIbo, it' Bhoiild be noted thnt the fitudlea c^rtt^ftln other intercBtlng 
ind helpful flfipects for planning Btudlea, For pxnmple, relative to 
era and Ufleei of studier, KoroHt ' a ('77b) .^technique of^ Identifying 



Btudy uB^^ra' criteria for Judging success pf the program is noteworthy; 

<^ > . 

and, the concerns by legislators and administrators which led to 

... * 

Goettlng's ('82) study are "^instructive also regarding users .and' -uaes. 

Forest and Marshall ('77) also identify a study user's (administrator's) 

. ' ♦ 

specific request as • the bapis for their study of an entire county 
Extension prdgram, • ^ ^ . 

A Before you inspect more closely the abstrac^F^in Volume II, we wish 

r'^^m.. • r ' • ■ ■ , ■ 

*t&:>r:emind you th>t these examples wefe jselected from among those avail- 

■i , " ..^ , ^ ,^ ■ 

^^Jff^t^^^j^^^^ previously discussed. . fhey^ajp;e ''not necessarily 

. ■ . ■ '^-^ '^^"^ ■ . ■ ■ . ,^ 

^ve^ "model'' studies, but rathi^ib as'*ex4tn^^ of studies. 



^ i*Bifij6^^ four study designs presentied; - The ' examples, show how 

^OT^^of^ Ext^^^ results using' these four designs actually 



r- Note also tiiat several '^^c^^^^^ studies, while grouped within a 

paFticular design, really lised more than one design. And, some of the 
studies which are grqiiped. within a partiq^ular Extension program area 
actually encompass program activities in more than one' program ar^sC^ . 
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KxtcnHlot< ptogramfl . rthaca, NY: Media Servlcea, CornoU 
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1»AUT TT1--SUMWARY Mij CONCIUHION 



,"■■■■'*■ ■ * 

I.fke an art collector aettloR out to clioo»tJ « work of art, you m 
.a prop.rnm evaluator, evftluation aponeor or* advisor liftv« iibundant 
choltcH* Reading thin readufet* publication ifl like browaing through 
.'an art gallery. You began in the gnU^pry'a ^ain ball and have panaed 
through Che corridorr. "displaying each style of art, exatnJnlng aome of 
the /pOHHlbilltleB, You atand, once again, in the main hall. It'n now 
• ci/r to begin your reconsideration of types of art In order to aplcct 

a suitable piece.. Remember that you may. find no single pic^ei? of art 
. to be perfect; the beauty of each ia in the eyea of its beholders. 

^ Li kewise, there nay be no «ingle heat de^sign for an evaluation,' 
^^^■kvaluatio itself an art (C!:o.obach, '82). Your choice or 

rc^^ommendation will depend j).artly on such factors as y9^^i'^ 
'^administrative, program delivery, or evaluation responsibilities. 
Your choice or recommendation also dej)ends on budgets, time aivailable, 
anticipated criticisms of the capacity and rigor of a design, and the 
political climate. As you plan for a study of program results, select 
a study design as • you WQuld a piece of art, fitting It to the 
Sfe 'and reso vlthin the situation at han^. 
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t-vNiIunlffit f^rmHUv anit hoW| upaiaca along tU« itna / fiom** of ^jChi?^^ 
v«lu0 JutlpunlH ^rti explicit s oth*ar« ImpllcUt 

Bi*itli» i^Hplt^itly, from th«? nrrtrr, <*tui contlnvmlly revl^wi^d, Sutii v^Uie 
)u<Jgmt?ntM will probably Iriflu^ucf* th** fltudy fliullngn ih^ ewnuiu}^ 
tn'rtluat loi\,^ of H progfrtnu Accordingly, w» adyUe «tudy t^ama to 
rt?cord thiUr premitiita Jind valu% JudBm*?nt8 In nc^liicclng And Impl^- 
*mi^itlng n ntudy dofalgn mu\ detilgh fiicetH, with a vltjw toward clarity* 
hoot?tityt rtod hoU^v^ibU Ity, ^ 



A 5>ummary of Stflrtfhg Points 



You may now be ready to choose or fortnulnttf HxiVdef^lftn In planning 
a Btudy of progrfim results, but pefhapR the many observ.itiona .you hnve 



'ade while "browalng" have lt;ft you *rather overvhe)imcd. So, at th£j 



Bk of over.slKplif ying nn individualized procpfl!^, wc offer you fieven 
starting points for selecting a dc^5i^^^^ to study ExtetJHion* program 
results. These startlnp points tend to follow the facets of* design 
discussed earlier. * 

The seVen starting points are expressed graphically below in the 
form of Jiif-ther?* guidelines. In a manner speaking, ij[ you view 
your situatvlon as corresponding to "A,'- nhen consider options "M or 
N." O^v 11 yVur situation is "B," then consider options "N, Y or Z." 
In guidelines one arid two, the "ifs" tef^ to Extension progfamrning 
situations. In guidelines three through six the "ifs" refer to 



rBtiulr.aiilk^^fO» tin em|c>i bt.iitHi>«* of J»r»>ari*» resui.ta, OuiileUoe btvart 

pr,c««ntttU tn ih« I'tiMt^r f,f»liw«,» Willi varlotia **th#n" Hptloiitt |>r«- 
anucti.t U\ the fifilu Li»li»«n, Th«»r# way h> occsstpnal «;i)<capft«n« t»» 



> \* ■ ■.; * ^ • 

/ ■■ * .* 

. •' • • V - . * ■ ' • 

• * K : 

The tlecln^m to rx/imlnc^ rt^nul t;4 I^Af],' (a) on- with 
clc.irly ^dff ln«-d groups of new piir t Ir Ipantii * /^r ' ^'t>^.^^^ jcntiiely n*?w 
%>r<^Rr.im to Implement lul ^iHov% luu.- u I /tny ttu> J^^iur ntudy UeH ^ 
^^clivfk'F: oi\ results within thcru* two pri"'pr**Ru^ing Hltuattons c&n b»* 



achieved throu>^h llie t ihe-f.e r Ir^ » c'^ompciriHon group and flcdd t-xptirl- 



l(roi 

n.-cnt; rhor^e throi* dtvilnns ri»qtilr<** t"^r *^ilow data-tu brvcnllectrd brforc 
ami .li ter prof;r.ir. Inph-mtntat Ion . 
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Figure 6.2 5 Centralizatiph of Program pi/rection 

» <5 , , 7 ' ' 

/survey 

If centralization t)f J/^. 

program^dlrectl^n xs^:^2— Moderate . , / Tlme-?erles 




Comf>arlson group 

j"^ ^ Field experiment 

' ■ , , •■ ^ ■ ■ ^ ■ , 

% ■ . • / " ' i ' 7 ■ 

Studies of the results of p^;ograms which are variable over time- 
and by ^geographic site are gene.raliy mo^fe feasible through the survey 

'.design. " Such variability in specific program objectives, .audience and • 
delivery methods occuris When Extension programs have decentralized 
program direction, (e.g. county direction of programs Is emphasized 
rather than State direction. Tl^'e flexibility of the survey design 

' permits ^t to cover a broad, range of program eve-nts and. consequences. 
Subjectivi'st study methods whi/h are frequently employed In the survey 

* design permit aggregating ^pecific program— results of a variable 
nature into more global categories of progr"am results. 

. ■ V . ^ :. . . I . ■ 

' . Studies of the ffesu/ts of programs with mor^^£entralized direc- 

t ion 'may use the ^flie-series or bfetween-group designs. Greater 

' /' ■ ■ ' ' ■ ■ ' • . - ' 

centralization of di/ection usally means more standardization of the 

/■ ■ ■ ■ ' / ■ ' ■ ■ ' • ■ 

program across state, district- or county lines. And, progpm standar- 

■ ■'/ ' ' ' '/■'■* • ■ ' " 

dlzation aids objectivist methodologies such as highly • quantified . 

■/■■.-■ /' - " . ' ■ ■ - ■ 

measjirements of/ changes in clientele • performance/status which are 

typical of tyime-serles, VGouiparison group and field experimental 

/ ^ • . ■ 

studies. Hifeh program standardization also contribute greatly to the 

. formation , and use of comparison and/ov control groups. Programs which 

are autl>/rize*d and fundec^^^holly or primarily from federal or state 

1 eve Is/are typically m.ore standardized.. . ^ 

: /.- < . ■ ' . ■ . ■ : 

/ : ■ ' ' " ■ ' 97: V • ■ 



Figure 6^.3: Degree, of Certainty Needed Regarding Program Results 



Survey 



, Moderate 1 

J ^ ^""^ — --^Time-series 

If .degree of ^ ■ ry 





certainty required^ ^ . \ » , 

: t j ' .-^ \J Comparison-group 

by study users xs ^ 

^High — " ^ 
^ . ' — — — -Fields experiment 

- ' "■ \ * w» . ^ ' 

If a significant evaluation audience has demanding, standards for 
proof that an Ext^ion program caused or contributed to clientele 
behavior or status. ' the evaluator should consider a "between-group" 
desigt^ -i.e-.a comparison group or field experiment: These designs 
are particularly effective in proving that changes in people- or their 
envirc^nint are due ;il or/in part tp an Extension program rather than 
\o cSf-ir .'factoid. However, the survey and time-series designs' are 
'Uso capable of producing credible findings on Extension program 
results, depending upon audience Jor the study, instrumentation, aata , 
■ collection techniques and especially ^tatisticaV analysis and control-. 
Moreover, the survey , may be the only design: 'feasible for studying 
results of some programs. . 

Reliability of evidence on results of an Extension program may 
have implications for the. way in which the • Extension program is 
conducted, as well as/ how it is evaluated. Study , designs which 
provide "the higher degrees of scientific certainty regarding an 
Extension program's (vs. other factors') results also pose constraints 
/on how the program is conducted. For example, a bfetween-grou^ design 
f may affect program 'timing, program cost per participat^'t and location 
of program* implementation. . ^' - ' . 



Figure. 6. A: Study Resources 



.Lqw^ —Survey^ 

If study ■^^^-'^ * . , 

resources compared:^— ^ Moderate-*- • Tfme-serles 




to scope are ^^"^"^^^ I 

— High ■ — -"CpmparjLson group 

-Field experiment 

' The questions of how many dolla^^-s and How much 'staff time will be 
needed for e^ch of the study 'designs under consideration must be 
addressed,' In general, the survey design Qosts less^per person (or 
group) covered by the study. Surveys based on retrospective data 
regarding progW re^lilts may use relatively simple (and thus le^ss 
costly-) statistical treaWnt of the data. ° Time-series studies 
require more freq^uent data collection than, surveys afRT thus use more 
•resources. - , Relative ^ to the time-series -study, adding data , from 
comparison or control groups for the . between-group designs^ also 
increases the amount of data to be coll^ected and analyzed. ■ y \ \ 

Concerning scope of programming examihed* for restilts, the "survey 
Is likely ^ to provide broadest , coverage. However, the evidence on 
program results from a', survey 'Is more likely to^be based on client 

.perceptions. , . ' ■ / 

■ Program evaluation expertise Is also a resource tihat may vary by 
study design selected. ^ We dQ not assert that 'some study designs 
necessarily require more, evaluation expertise than others, bfut rather 
that " study design selected must match professloral Interests and 
competencies of staff available. • . 



Figure 6.5V Study Time Period 



If the time period 
available for study 
completion is , 




Survey v . 

Time-series 
Comparison group 
Field experiment ^ 



• Time period. available -.for the study depends on such ^factors as 
how soon 'documented results are n^edW, and hqw ' long those conducting 
the study will be' available. The ^survey design lends^ itself to 
shorter time periods for study cdmp let iba. The r^trospective nature . 
of the survey makes it unnecessary for tha study time^jfriid to be as 
long, as the time .needed for program effects ti occUJ:. This Is " 
because, with retrospective data, multiple data collection passes are' 

not employed. v , ^ ^ 

,If more time is avalTaBle for study completion, the ^e-serles, 
comparison group and field experimental designs are favpred. In these 
thre^ designs it ip, ^^sually^^cesgary^o'Vit" fot program results to 
occursq that data may be ' coll'ected' to^*sie^f\ they do i-n 'facte ' occur . 
The time period needed .. to cotCiJuct^ a time-seiles or between g^oup^ 
design may be briqf if the study 'is. sgeks shorter-term rather_than 
longer-term program restilts. And, if data on clientele regarding the 
"before" time period are already ^available, the exercj.P6 . of t^hese 

'designs may be hastened. , • 



1 

Figu'te 6.6: Scope of Study 



If the scope of 
the study is 




Wide 



Narrow 




Survey \, - 
Time-series 
Comparison group 
Field experiment 



Wide-scope studies.'^^^h in terms of geographic area and program, 
content, or subject-matter, ar,e easier to achieve through the' survey 
design. Th^^cost per person or group is lower in surveys than in 
btW-d^nfes, allowing for a wider-scope study with a given amount oY 
s^idy resources. ' * 

Tl\e generalization thfr£ the survey lends itself to w^de-scope 
studies is applicable particularly to .studies that^ examine a wide 
subject-matter area/ The data 9^ I'lectio^ instrument of a survey c^n 
mo.re feasibly ^ cover a broad range of j/rents and consequences. The 
specific, ^quantitative comparison typically pursued in the time-' 
series, comparison ;group and fieiTd experiment encourage studying the 

results of programs \Jrith i^arrower subject-matter content. - Jc 

' ' "\ ' . ' ^ ' . ' ' ' ■ . • 

Tim6-series arid between7group^ designs -are more manageable,, tor , 

' ■ . . • . * ' ' ' ^ .'^ 

examining results of narrower-scope programs.. These .designs generally 
requlrldnta to be collected several -Wmeffi. The aollfefctlon brings- 
examiner In coniiact more with delivery of the program being studlct^- 
However, It is feasible and may be desirable tp> conduct tkme-serles 
and between group studies over wide geographical areas. 
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Figu4 6.7: Coordination of Program Delivery and Data Collection 




Independently - . 
'from program — Survey 



ERIC 



If data -are colleeted^ ' JTj.me-sexi.es 



In concert Ccmparlson group 

with program 

' ' ^>»Field Experiment' 

The survey\ allows data to be rcollected ^ndependejitly from the r 
delivery of the program while the other designs' generally require data ^0 
collection to be ill concert with program implementation. That is, the 
timing, personnel dtxd purpose 6« a surve^^tudy are not tjece^arlly 
'^closely associated with the timing, personnel andj objectives of the 
program ^t^ed. A survey can be aonducted long after a progr.am is 
underway, by individuals not known by : the program staff and to 
measure. results, not necessarily matching .those fought by the program 

■ ■ ' ■ ' ■ ' ' , ' ■' . ^ ■ . '■■ . . ■": 

staff*.,' ■ ■ / , ■ 

On the contrary, the' time-series, comparisxin group and faeld 
experiment designs myst be more closely coordinated with program ■ 
planning and deliVeary. .Timing^, personhel and iresults to be achieved 
and investigated arc interdependent/'. Especially v/ith the betweep ■ : 
group designs, both program planning and; how results, are- ,to<. - 
examined must go hand-:Ui-hand. Where,, when. • how. and with whom the 
program will be implemented and.>n|t; imp.lc,tnented . is . of mutual concern 
to both program staffs and e.val4tiop • staffs. >r to cacjvof tllese 
roles if the' same pj^^ns^bn perf(ffnel ^r^e r^potrsible for U)th projjrnm 
.delivery and documentation of jjrogfam 'j^esults ^ ^ 
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Summary of Text * 

\ . . . - ■ 

In Part I\ (Chapters 1 and 2) we provided the background nnd 
approach of this publLatlon, along with an overview of general- 
fuidellnes and standards ' for progranf evaluatloris. . We also descrlb^ 
the procedures for obtaining the examples for Vqlume- T of this 
publication. I.e., the 42 abstracted stucf^es of Extension program 

results' In Volume- li; Two social science research firms with 

.■ ' ' ■ 

experience In assisting ^ federal and ^ state agencies in program 
evaluation established" a\ pool of 148 positively appraJs.ed studies of 
.Extension program results, from the period 1961-1978. Using tile 
procedures and technical ac,cvrracy criteria developed by these firms,. 
ES-USDA added five studies fron^ 1979-1982 to th^ pool, for a'total of 
153 studies. The studies in Volume II were sQ^ected from the above 

" • • 

pool of 153 studies, 

In Part il (Chapters 3, 4 and 5) four study designs for (examining 
Extension program results were described along with examples of the 
application of these designs. These examples show feasible ways of 
uncjert^aklng stuc^ies' according to the four study deaign^^. Wc 
systematically identified ' ten facets of stuJIy designs to hlustrnte 
the tunny options open to you in formulating and implementing /a kudy 
design. We cited examples of ■ these options nt v^k, within each of 
the.fbur designs, among the studies tibstracted in Volume 11.^ Finally, 
we dlscuss.cd issues . regarding implementing eacft of the^deoigns. 

Here, in Part III, we invited- you to reconsider somo of the 
starting points from" which you migh^ decide? on one or another of the 



designs. We* acknowledge that you may conclude that some other 
approach to describing Extension • program results^ suits you better. 

9' 

Perhaps further .exploration outside the frameworl^ of the present 
dijsigns* may .be in your interest. Ho.wever, if your interest is to 
develop a study based tfn quantitative i analysis, thi^ publication may 
scrve^ns a resource. _ ( 



The starting points .from, which you might choose one of the four 
designs involve "if-then" generalizations, to help deal with Ideal and 
prac^cal considerations relating to selection of ^a ' study design. 
Th^e "if-then" guidelines go further, than maoy writings in the field 
6f evaluation In systematically setting forth a "situational approach" 
to selecting a study design. For , examplt , Campbell and Stanley (1966) 
confine discussion of the criteria for selection of a design mainly 'to 
how much certainty is needed regarffing program results. 

However, Cronbnch (1982) broadens' the set of criteria for 
selecting , a study design to iriclude ot^hcr factors, ,such an needed 
scope of a study/. Relative to selecting an overall approach to .nn 
evaluation, ForesU (19|!0) and Forest and Boyd (1«^?) Include such 
factors as the or||nizational level at wliJch infom^^^tion on program 
results ifi needed and potential for clientele . Involvement in titudy 
proceduroH.j Patton -(1982) ^rclntcB the situational approach to 
planning 'and * conducting program evaluations to thq standards f or 
program evaluatlonn which arc suggested by the Join* Cotimilttoc on 
Standards for Educntlonai^ Kvaluntlon (1981). ^ I.e.. utility, 
fons^lbllljty, propriety and nccuracyi 

11)0 ■ 
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CliAl'TKR 7; IN CONCLUSION 

; . • ' • \ 

Wc hope that this publication will assist Fxtension program 
lenders and specialists in selecting or advising on the selection of 
study designs and design facets. We feel that the choice of ;study 
design for ascertaining results of Extension programs is strongly 
relevant to the role of Extension program leaders and specialists. 

An evaluation study's quality and relevance (to program staffs' 
responsljbilitics for ^progrnn development, management and account- 
ability) aro greatly (\ffectcd by the choice , of studj^design , Ve feel 
that program leaders and specialists have much ^o contribvjte to choice 
of study^dos^n without becoming specialists It program evaluation. 

Dimensions and Steps of Ijrogram Evaluation 

■ ■/• • . ■ ^ 

lu plnilng thlr, pubUcatlcMi In piUfipcct » vr , we havd nnphanlv.rd 
thnt evaluation of program results is,^, so to speak, "only one piece vi 
the evaluation*. pfe;''_ evaluations which focus on several aspects of 
ExtohMlon programming^ other than program results should he duly 
coUfildered In choosing where to expend perennially ncarce r'V;onrceM 
tor formal evalu^tl^iv (Ciroris, 1977; longeMt. 1975). Kurlhermorp, 
evaluating program rerailts requires oxpertlno, far beyond that on study 
design^ lucUuJtng expertise on: managing the i^tudy effort;^ deciding 
who needfi to ku<\w what Informrttlon through invob^lng rttudy uner« in 
eritaUYlnhlng erlterlrt for evaluating program" tenuUft; data g^tl^erlng, 
delta mmlysls and interpretation; and reporting* dlesGmli^nt Ion <tnd 



?"W3 • lot) 



utilization of tho ntudy f indlnRH. i-oncluslons .unl iccomncMidnt lonn 
(Byrn, l«.mS: Fronch. 19H <; IMtton. 1978), However, when consider h.p 

those other steps In the over.ill cvnluntlon process, readers should 

) 

recall that study design 1h Intiir-.itely related to every other step. 
That i,s. early steps In pmnnlnp'' f or a study of prorrnm' results will 
affect choice ur forrxulation of n study dcslRn, which in turn will | 
bear upon t{ie study's findings .ind utility. 

Appro aches and M<;tho da^lop.i''s - 

The "study deslp.n matrix and facet analynio In thin publication 
avc pr<>;ented ns frameworks to help with study deslK." selection and 
formulation. We feel that the deslj.:n options presented can fnclHtate 
quantitative studV designs that maximl^.e certainty rcRardlnj-. . proRram 
results, consistent with realistic prop.rammlnR and evaluation situa- 
tions. The fouJ designs iind the ten facets idcntlfl.'d are In no way 

meant to lb>lt design opinions to ^m-/ employed In the luture. 

A 

mcntUmed previously, wo do nut Intend to -li-U'ly that <;u.mU tat Wr 
approaches are to be ip,nored In evaluation of Kxtension proRrans. 
Nonquantitative approaches' arc simply not .wlthlN lhe..u-ope of this 
publication. 

An ritated earlier, the Mludlen ahtitracted In Volume tl of this 
resource do not necemiarlly repter.ent either all t'c'cent , or tlu' hont 
Of rtdc.,u.ntety conducted .judnt Uat Ive ntud leo of {.Kxteuo Ion proHf-im' 
remit tn. We mmnht variety In typer. of evidence cnncetn}n« the 
Influenre of i:xien«.tn^, proKn-unrt on cUr.Uel.- hrhavlo, or rttatun. 
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^' While .ill tho .ihstrnrtci! -itudicr. in Volur^r II rwi the 

rritt«ri.i *icccpteii tor tcchx^icnl accuracy, ?u^rr of the •;tiu!iifi havi* 
rxccpt ionallv hif.h conccptu.il clnrity avA oxcnplnry rr thodology . 
SoviMm) c^; the stiuliivi s!u^v crt\'>tivi" connrciions bt*tvr«-n • Innovat i vc 
Kv^tens progr.irr 1 n^: and ri^torrur. cvalunt ion ncthodology, Crrerally^ 
t!if stijJiv-i well tU-.l^rrd to bMvo achifVed thc^r Ir^ncdlnti' objcc- 

tivr?., ih.it of . rrrori iiiK rr'uili:i of >i«-lfrt<'d, KKt^-nnlon prop>/tmn or, 
ihv cn-.tfi and rrNuItr; of alt«Tnato noilns ^^^^ df^Uvrry 'rf thrir 
propj an*; . 

Whilo wr Intnid for t h I r; jni!) 1 1 <'at l<it^ to- be u?ic(ul Ivr dr?jlKnlnV: 
TitatcvMdo studitv; of Kxtrnsl^on |>ri>^:rani r«JttltM, vr wrrr nhlo to fif:d 
only a t«*v nvK^i .tudic-; a?: r >,;irip Ir ?i . flowrvc^r.' wo ror;t r>*udy 

ru'thcH!(^IoKlon oxrr::pl 1 1 lod lt» Voltino M o.in omplovrd ar4 fUartlnp. 

pt>lt»ts in do.iip.nluK -Uatowldc. ntudliMi. Althou^li thr ralorltv i^f t!u» 
.diMtr.iotfi 111 Vi>lutnr 11 loCH-i ot\ ooo pii)K,r^*m wilhin a pro^* r.m ai ra , 
f:t:iidiorj o! (ooprralivo r>.trtirj<.n ptt^Ktam icr-aihn nerd not. tir llmltrd 
to ft Ttin^lo r-ubj^Tt natter or program area. 

Tlu' .^>;amplr rituillc". luVolnr-r" n provldo ltark>?roiind ViuwlrdRt^ for 
cfc'ilHninK httipV/^t i vr . tuu ;ilra» r xp« r ! ^.nr c^d-harjcd , riUidirri on prnp.r.-im 
tfieiultn. |^'.^d«Mri at'' rnrnUraHrd to Unr th«? rKnnplf* r*Mtdlrri\^ OOUrf^^n 

^ nl td»^^i'i rathrr tlian tnodrhi {o hr rnpUratr-d. Wc^ In no vnv winh lo 
plao«5 pnrarti<^trr?i ^around futurn |f*rrort« to rvalnat«= <:«w,prf Jii4 vc f xien- 
rtion prof?ramri» 



. Past as Prologue 



' Finally, we refer to ;some^ early perspectives ,0^^ 
.r^.^ults c^ Extension's educational and information-transfer programs. > 
While formal evaluation of public programs has soared in the last • two 
decades,' -it" began early:> in Extension. The spirit and philosophy of. 
Extension work has always encouraged systematic study^^f program 

ef f eictiveness. . ' ; 

- .Nearly sixty years > ago, the U.S. Department of Agriculture - 
published, "The E^fe.ctiyeness- of Extension in Reaching Rural People" 
(Wilson, 1926). Wilson asked in this early, four-state study: 

—Is Cooperative; Extension really reaching large numbers of rural. 

people? . f 

^^ ^j^ h^ttf^r methods of farming and homemaking have been 

^accepted'as a result of Extension teaching? 
—Which Extension methods have been most effective in obtaining 

the adoption of these improved practices? 
.—In the effective conduct of Extension work, how important are 
land ownership, sizf of farm distance from the Extension 
r ^ of f ice,, mejnbership in- the Extension association, contact with 
Extension workers, and participation in Extension activities? 
' -Wh^t do farSts and farm women ^hink of Extension work now that 
Cit has begnle ^stab'llshed in large number of counties? 

. While the specific scope questions may appear dated, 

reclst in 'a more general ^£tug^^^^ require answers today. 

Thire is "indeed prologue i^i-the pa^t, knd if deserves our attention. 

\- ■ kk . ■■■■ -X,.; ■■ ■ ■■■■■ ■ ■ ■ 




For eAampXe, Wilson's and Gallup'9 (1955) comprehensive , synthesis o^f 

studies' 'of Extension profiram effectiveness, an Extensipj;^ classic, 

'. • '■ - - ■ ■ »-,•'■'• ' ■ . ■ 

reported impprtant generalizatior^s on th^ relation between .alternate 

rogram delivery methods and clientele use'of recommended agricultural 
and' home economics, practices.' \ * . 

More '^\roadly. Price (195?.) in The Spirit and Philosophy of 
Extensio n Work , called for five lines of investigation which appear to 

. . 0 ■ 

\ ' ' ■ . ■ 

be entirely relevant to today ' s situation. Tl^ese five lines of 
Investigation concern: ^ 

*(1) the philosophy of the Extension movement; 

- • ft ' . 

(2) the status of those by whom Extension Is provided; \ 

(3) the "Content of Extension programs; ' 



V 



(4) methods for planning, -teaching, and evaluating Extension ' / 

. _ ' ■ ' ' ' ■ ■ ■ -■ / 

' programs; ~ / 

(5) administration and supervision of Extension programs. ^ 



Numerous studies and publications from each preceding decide of. 
Cooperative Extension's history reflect the concern with understanding 
more about Extension >i)rogram results. Thus, we carry or/ a firmly 
ensconced tradition: the desire to know what effects programs have 
had, are havin>g, or ^ay have. In a spirit of c6fitinu^.t/w^ offer this 
publication, and, we hope *ttf^t it is acceRtdd withiij/ a philosophy of 
mutual commitment and sharing.. 

/ 
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Appei/dlx A! Procodvvres' for AbBtractln^ and App rniHluK StudtoH , 

7 In i>nr^lal ronponse-to n ijinndate, in- the C9nRrefiB':f onnl Food and 
AgijlcuUure Act oM977 ,. for%An ^valuation of the conaequonceH of 
Cooperative Extension program^ a natloiWldo search was^ cohd*Cted for 
analytical studies containing findings on Extension p^ir'bgram inipactH. , 
Some 450 reports from this search appeared, upon 5rl^f Inspection, to > 

• . . • . ■ . , ■ _ . 

contain evidence on resuUs. from people's partlcli>4ion In Extension 
programs dui^lng the years 1961-1978. 



In September 1978, -contracts were awarded to^ Kappa Systems, Inc., 



and :(KSI) Science Management C/rporatlon (SMC) tp'^itj^'te**^^ . 
^and. summarize"- these 450 studies. An advisory group bf Cooperative 
Extension "personnel from each region of the United .States was actively 
Involved by Extension Service, USDA, 'In employing Kappa Systems, Inc., 
and Science Management Corp. and In oversight of ' their procedures and 
report preparation". Two officials of the Budget and Program Analysis 
office of the U.S. Department of Agriculture also advised ES , USDA, on 
the. contract award to. and procedures used by. the two social research 
firms. Evaluation and program' personnel from several state Extension 
Slices also reviewed the final draft repojt and suggested revisions in 
these research contractors' establishment of^crlterla for technical 
accuracy of studies of Extension program results. SMC dealt primarily 
with studies completed during. the' I960' s and KSP with studies completed 

during the 1970's. , ! • 

In the judgement of the contractors approximately 350 of thej'450 
studies assigned to them £ ctually meet the criteria of 
(a) contained data on pr6dram results rather than just program 
processes; (b) focussed on programs clearly identified as Extension 
programs or specifying Extension' s. dontrlbutlon to a joint agpncy 
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proRr^if ana puppl Iml'^by j)ro8ram j>Hrtlelp,«nt« or cUant^^le 

. p«ph »Viulv r^VM meeting -Che minimal guaVifidaaonff abeva wgfl • 

daHcr.iptlon of the 
, pr^Rram 0lt#neU. thS intiChodologiivil approach employeil by the . 
^ ^iS, Sm^o f liullnpa and any coKclualonn drawn^ «bout tha program 
/YmpneW#py the tJfudy: author (a) . •31^"«iJn:2iymZf!^^^ 
#nnd-90pUHlona nre'lncludcd I n thu ^.batracta. ratheji_tharu^^^ 

li^jlp M^ ""'^^^ a whole . ^ 



/ 

//' aniT 

i:^. , ,, . ^ 

Api^^Hata werp' nuide as td whether eac/ study'* a flndlnRS ami 
,c^Mfi?n8 werc"wnrrnnted;" i.e.. whether there waa adequate 
^|^(.tlatlon accordin-R to d«ta pronented In' the repoc? and reaenrch 
%^^mlo\oly/of the author(8). The appraisal criteria ItK-luded: 
^^^'4vcon«*8""^y °-^''«'="**y hypotheses and- measuremontH with 

i-'^ objectives and structure of the Extension programs studied; 
b) validity of measurements and 'analysis of the data; 
^) basis for any inferences regarding rfesultn of the progrnm 

f 

Studied 

d) basis for any generalization of sample findings to a larger . 

population ; ^ 

e) primary da collection from clients with-* adequate size of 
• ■ , t ■ . 

sample and response rate . , ' 

One hundred and forty-eight .study reports were found by Kappa 

• ■ t 

Systems, Inc. (1979) aiid Science Management Corporation (1979) to 
contain Extension program result findings and conclusions which are 
warranted relative to evaluative research criteria. The 148 studies are 
reviewed as a somewhat representative assortment of externally verified 
examples of program^ result findings across Extension's program areas. 
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Forty-nil^ study* rephvtts 4>i«lHi\ed to KtU fur review «iut dpjir«teal 

»• . ' ' ■' 

w*jrn noaiKneii hIho to SMC. An fUialyals of e ImiUrltydlaalmiyarlty 

betweon t|ji4* 'two contr5tora' ruv lews and apprfltaalg was baaed primarily- 

upon jhe foVlowinR queations To what extent did both corapantea agr^e / 

that \l>e llndinHM and cfnu'luHlona of ench of the A9 reporta «re 

Tlu» two contracto^ nproeil upon 71% of ih«* Btudy rt*port.ti (35 of the 
//) rc«porCfl) an to whetbrr fiiuUnRa mM\ conchifilonQ oo px^tm r^aulta 
were Hufficiently warranted by r^tiuly methodology. Considering only 
major dlfft?ronct;H betwoon concractora' Judg«:muntfl on l\w (|uttlity of 
HiibstAntiatlon of fV^dlnpn and conclusions regarding program re&ulta, 
contractors agreed 92 percent of the time. 

The mnltt-volume reporta by each of thu two contractora are Hated 
below. 

• Kappa Systems. Incorporated ^ 

. Classification of Selected Extension Studies with Impact Findings, 
Vol. I, Kappa Systems, Inc.;, Arlington, Virginia. 

Extension Program Impact Flndltigs from Selected Studie s Conducted 
from 1961 to 1978 , Vol. II, Kappa Systems, Inc., Arlington, 
Virginia; and Appendix to Volume II, Supporting Abstract. 

Guidelines for Improving Extension Program Inpa ct Studies, 
' ydl. IH, Kappa Systems, Inc., Arlington, Virginia. 

Science Management Corporation 

Inventory and Classification of Reports Abstracted , Vol. I, 
Decision Studies Group, A Division of Science Management 
Corporation, Washington, D.C. • >^ 

Final Report: ' A Review and Appraisal of Studi es of Extension 
Vol. II, Decision Studies Group, A Division of Science Management 
Corporation, Washington, D.C. 
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.TIilH attaelimtiut ja deaJjneji^tOihelp rh« peraon aehlgnat^d by the 
Htatd'TrirV(:cor or AifminiaFrfttcrr ttT^ Tind and Jg^leet a enwplete act^dj?. • 
ranort .-o^>ra In-lngaiut in£BonJ^)<rettalo» yr^gram MagCa* reply to 
FstuBVA'Tl^i?^^^^ t)0cewB'erl. Hft2. ' As explained in th« 
of rfqiieat CO the Htrtte nir»n;lofa and Admlnlfitrfltora , our tuimmtlate 
Intent la to odd to ft pool of tsxaiiipl** fitudttja trow 1961^1978 with 
^.^amph'H from 1979-1082. Jha Unlveralty of MaryUnd'a Cooper«ttvP 
K)(t«nfiton r..»rvlc«/Uepartraont of AprlPMUurftl and I'.xtenaton Kducatlon 
will then n»i1e(!t froro thla pool the jnoat appropriate examplpa for • 
Inclusion in a guidebook for atale atudlea of program impact. 

Aft you pfarch for arid atilect a acudy to forward to KS-USDA. keep 
In ralnd that we methodoloBically adequate 

relativ e to atfTm lnrd ft whiTIt wure' fo rmulated and employed In _a^jmat. 
•j^il^nd appnnCTTf' atudiea on Txtenaion profsram ef fectlvenea^. 
Wd are aware that it may not be IToBfilble" for some KxtensJon organlsa-. 
tion« to locate a otudy of their programming which meets the criteria 
H«t forth below. 

A full cxplanntlon of the processes of review and appraisal used 
by Science ManaRcment Cor p (SHC) an d Kappa SyHtcma. Inc. ^ K Sl() la 
nrenented in Volume III of Kappa System's contractual report, ^'Ui^e- 
Hnes for Improving Extcnaion Program Impact Studlea,' July 1979. 
Tbla volume, along with the oth«>r volumea from the contractual work 
to review, appraise and summarize studies from the 1961-1978 were 
nfll lcd to State Extension Services in September. 19T9 , as part of the 
1978-1980 Congresslonally mandated evaluation oj ecdnomic nnd social 
consequences of cooperative extension programs. 

This attachment has been prepared to provide you with enough 
information on the type of studio's wc are looking for»3riO that ^oujlo 
not have to rend the Kappa Systems. Inc.. m aterials to respond to our 
request . 

' Stu dy repoxts may represent research from academ ic departments 
Mnc luding theses) as well as studies sponsored of conducted by your 
.. Extension organization. .Comparison group, b efore-after and survcji 
studies will generally be most useful to us . 



*Volume II by KSI, "Extension Program Impact Findings from Selected 
•Studies conducted from 1961 to 1978" references all IA9 studies 
selected from a larger pool of studies by KSI and SMC as containing 
impact findings and conclusions with adequate substantiation. 
Abstracts of these 149 studies may be found In KSI's Appendix to 
Volume II. and SMC's "Review and Appraisal of Studies of. Extension 
program Effectiveness," Volume II. 
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^»leati*» a«Uci rtiul rorwar^l *» titway o* y^Mr t'»»ott*i which, in ywv 
nptnton., meets or t\d|ltty^H)^^^ 

InFnf aatf ^. if it 4og» iioe eVhfsio f in4lpgs on iapiets o« 
^ ^ n t en h i i»u pro g r aw ^ 

» atudiea B houhi eo Htaln__f tn^iBM wlthtn aLiigf^ «^ 

iVeToriowTrrg thr^^^^^ progf^w ifapftctai 

(a) L**«L"i«*J. (Mtsnsion program irrtnifipant*- (e, g, , 
VnWlatjge, attitude. e^tU or aspiration ehapge) J , 

(b) practices or aj ipj jca tjtjin of learning by program 
pHTtTcTpafU^aj' and / * 

(c) conaemiiencea of parClctpania* learning or prac- . 
TTcea --^eonoraic. boetal, paraonal and/or 
environmental, tnt-ludlng benetita, aatlsfacllontt, 
n@ed^ or prubleraa. 

. Impact data_«hmUdJ^_coyi'£^^^ "'"^ 
only irirm KxtVntilon ntaif reports. 

• cllenta' or oba0rv.'rs' («.R.. ''.-Hera' part-nta) p«rcep- 
tiona of impacts are /icceptable aa uell na objectively 
mi'aBurt'd Impactu. 

• ^tMtidi&a of tliwnteUi'n ricureen or rating* of valvie of 

'information (Including l-xfcnaion* w) may not contain 
impact findings (ficc example No. 6 an appended). 

2. ' The fltudy- report mU-c^tm tcly doneribea t he Kxtension program 

being assessed. That la, the report IdYtlfiew. <^-g-. . 

• need or rationale for the program , 

• program object ivcn and how they might be accomplinhcd 
-through program activities 

• Extension's particuLir programming contribution, in 
the event that a Joint agency program Is evaluated. 

3. The yeport expresses the purposes of the evaluative study, 
in relation to: 

• the objectives and structure of program being 
evaluated 

• rationale for the evaluation— Intended importance and 
utility 

• Identification of audiences for whom the evaluation Is 
Intended. 
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luiiiuUrivm r^if |>r*'iira»4, .iM^nr , H^utciU part u 
ti4fi4Ui in Urn iH^J^i^w 4aa rli«i bampUng tu^^^^^^^*=^» 

r 

^ * 

\iSiB|)rt.n» or Muifa "J dUdlysU "tt^t'i titan iiuUviduala 
bvuh aa ftitas, orgaHla tons 4n4 gov^einMiiHral uhite). 

ci)i^ent or eKputtMic to or htvulviJmcni; to program 
iliiUvtiry 

*4Ci'ounttJa for client ouceota^ (cU^nlti* tttiU- 

tiut^ tt^ K?sC«^n?*ion progtWiert nt^ itcc^rptnbU) . 

The Htuily reptni^4Ucu««ieri the vallcilty of the m<>iiHurej a£^»iCH 
or ob\ii rvatimlC of rlie^nt t^ W UarT iinftt prae t ica a_gjn^ 
c7tilt^ qu^m^ oY i^artUnp/pracrto^Ht 0*gt» th^ report: 

• nhown that the 4n«t rntPc^nt n or uhn^rvntloti p!i?;irturt^ 
varlablen that uro rrjcvnnr .utd appropr l^ite 

• shown thrtt the lufit rum<3nt n mivquately cover Ch^ 
iloninlnH of vi^ll speciftt^a constructs. 



The stucty report's j ind conclutiionH nppe/tr to )ic 
hi\n vd_on i\ valid iHu ayHTft ol the r eT^itxi^ the impnctgi 

of the r.xtension proKr^imt * ^ 

• loRlcnl rt^lntionship?! nt^ t-stabllnhcd between datrt 
5et 

• adequate Inbi^l ling of tnblcB. charts nnd graphs 

• clear sepnratlon between findings based on data 
collection and analysitw and genera) conclunlons about 
the progrnmn' ref?ultn. 

The study report provides a comparison If pro f^ram succcfi« 
or f ailure is judged , e.g.. 

— ~ ; \ 

• program impacts are compared to some established 
standard or goal, or to impacts of other programn of 
similar nature or to absence of a similar program* 
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<Uit) 'l4S-6fa:s it ia |ir^4Vriibl*3 tin tY^opi?rs»nv** ^HC^^oslPu/^'^^UA oiair 

Heviue^ta to the UM tut photaituia tt area ahciuld Cintlain NAS )^ 
nusaht-rn asjst^ucd to tht^ r€f|HHff> which ^tc Jcolr^J* Hc^qurttti* hcaf inft 
t".tl! tuij»hf*rri iinrUMir»l hv thr N.iiionnJ Agr 1 ru J t ural l ibr4ry vllj he 
rx|ir«Hl«^a. <:4U mmhrrri mAy he itht^lned through thr AXU^ssuit^^ rrtrJrval 
f cic*rvUi^. ACKlO'tA f Afitltultutrtl Onltn«? Access). Thr ACHtCOU 

unlvrtrtit U/^ aiuJ at mont UniJ-Rr-^nt inrtt Hut lonH . 

rutl rcportr* ot thc^ sfAdirn citt^tl in thiti ttnoMVc^ publlcntinn aay 
* be used 4t the UcUhv lU*' locnt ion the NAL (n«?nr the intr r »«!cC ton ot^ 
Route I .ind Intt^rnt/itf* Route '95. iibou? 15 faiUf* northeast V4^htnitt6n, 
DC) or 4t the Dintrlct of Colunblii Rrnnch df HAL (Room 1052» South 
Building, VSbS, Uth ^nd tml<»pendcncc Ave.. Wnfthington. DC 20250). 
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